2024-09-21T15:02:59.262 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T15:02:59.277 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T15:02:59.360 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971 branch: reef description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/rbd_nbd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7909971' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi no_nested_subset: false openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: extra_packages: - rbd-nbd flavor: default sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: a477b1894d5c5a613f23260ca0e6b96d79652ba7 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 4623 sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 sleep_before_teardown: 0 subset: 80/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: a477b1894d5c5a613f23260ca0e6b96d79652ba7 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_22:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T15:02:59.360 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa; will attempt to use it 2024-09-21T15:02:59.361 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa/tasks 2024-09-21T15:02:59.361 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T15:02:59.363 INFO:teuthology.task.internal:Checking packages... 2024-09-21T15:02:59.383 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9' 2024-09-21T15:02:59.383 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T15:02:59.383 INFO:teuthology.packaging:ref: None 2024-09-21T15:02:59.383 INFO:teuthology.packaging:tag: None 2024-09-21T15:02:59.383 INFO:teuthology.packaging:branch: reef 2024-09-21T15:02:59.384 INFO:teuthology.packaging:sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:02:59.384 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef 2024-09-21T15:02:59.625 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-871-g6a1bdf85-1focal 2024-09-21T15:02:59.626 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T15:02:59.632 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T15:02:59.632 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T15:02:59.697 INFO:teuthology.task.internal:Saving configuration 2024-09-21T15:02:59.709 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T15:02:59.734 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T15:02:59.758 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 14:56:58.221685', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-21T15:02:59.781 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 14:56:58.219889', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-21T15:02:59.805 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 14:56:58.223218', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-21T15:02:59.805 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T15:02:59.811 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-21T15:02:59.811 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-09-21T15:02:59.811 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0'] 2024-09-21T15:02:59.811 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T15:02:59.889 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5dca0ff640>, signals=[15]) 2024-09-21T15:02:59.889 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T15:02:59.895 INFO:teuthology.task.internal:Opening connections... 2024-09-21T15:02:59.895 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:02:59.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:02:59.985 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:02:59.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:03:00.071 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-09-21T15:03:00.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:03:00.174 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T15:03:00.212 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-09-21T15:03:00.218 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-09-21T15:03:00.219 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-09-21T15:03:00.225 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Ubuntu" 2024-09-21T15:03:00.225 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-21T15:03:00.225 INFO:teuthology.orchestra.run.smithi067.stdout:ID=ubuntu 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE=debian 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="20.04" 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_CODENAME=focal 2024-09-21T15:03:00.226 INFO:teuthology.orchestra.run.smithi067.stdout:UBUNTU_CODENAME=focal 2024-09-21T15:03:00.227 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-09-21T15:03:00.254 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-09-21T15:03:00.260 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-09-21T15:03:00.261 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-09-21T15:03:00.309 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu" 2024-09-21T15:03:00.309 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-21T15:03:00.309 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu 2024-09-21T15:03:00.309 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian 2024-09-21T15:03:00.309 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-21T15:03:00.309 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04" 2024-09-21T15:03:00.310 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T15:03:00.310 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T15:03:00.310 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T15:03:00.310 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T15:03:00.310 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal 2024-09-21T15:03:00.310 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal 2024-09-21T15:03:00.310 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-09-21T15:03:00.329 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-09-21T15:03:00.336 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-09-21T15:03:00.336 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-09-21T15:03:00.383 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="20.04" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=focal 2024-09-21T15:03:00.384 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=focal 2024-09-21T15:03:00.385 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-09-21T15:03:00.404 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T15:03:00.413 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T15:03:00.417 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T15:03:00.418 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-09-21T15:03:00.420 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2024-09-21T15:03:00.422 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-09-21T15:03:00.427 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T15:03:00.433 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T15:03:00.433 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-09-21T15:03:00.435 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2024-09-21T15:03:00.437 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-09-21T15:03:00.465 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T15:03:00.536 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T15:03:00.550 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T15:03:00.550 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T15:03:00.551 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T15:03:00.551 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T15:03:00.552 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T15:03:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:03:00.557 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-09-21T15:03:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:03:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:03:00.559 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-21T15:03:00.559 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-21T15:03:00.565 INFO:teuthology.orchestra.run.smithi146.stdout:5.4.0-124-generic 2024-09-21T15:03:00.566 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-124-generic 2024-09-21T15:03:00.566 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.4.0-124-generic 2024-09-21T15:03:00.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-09-21T15:03:00.566 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.4.0-124-generic 2024-09-21T15:03:00.566 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2024-09-21T15:03:00.567 INFO:teuthology.orchestra.run.smithi067.stdout:5.4.0-124-generic 2024-09-21T15:03:00.567 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.4.0-124-generic 2024-09-21T15:03:00.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-09-21T15:03:00.666 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2024-09-21T15:03:00.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-09-21T15:03:00.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-09-21T15:03:00.889 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:03:00.892 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-21T15:03:00.892 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-21T15:03:00.941 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:03:00.941 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-21T15:03:00.964 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-21T15:03:01.002 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-21T15:03:01.143 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:03:01.143 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-21T15:03:01.197 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-21T15:03:01.277 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,227 kB] 2024-09-21T15:03:01.355 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-21T15:03:01.415 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,601 kB] 2024-09-21T15:03:01.416 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [818 kB] 2024-09-21T15:03:01.430 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [818 kB] 2024-09-21T15:03:01.430 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [476 kB] 2024-09-21T15:03:01.435 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-21T15:03:01.435 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,156 kB] 2024-09-21T15:03:01.460 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-21T15:03:01.460 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-21T15:03:01.464 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-21T15:03:01.464 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-21T15:03:01.469 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-21T15:03:01.476 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-21T15:03:01.477 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-21T15:03:01.478 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-21T15:03:01.478 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-21T15:03:01.482 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-21T15:03:01.482 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-21T15:03:01.516 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,037 kB] 2024-09-21T15:03:01.525 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [554 kB] 2024-09-21T15:03:01.531 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-21T15:03:01.531 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,283 kB] 2024-09-21T15:03:01.549 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-21T15:03:01.552 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-21T15:03:01.552 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [460 kB] 2024-09-21T15:03:01.555 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-21T15:03:01.555 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,232 kB] 2024-09-21T15:03:01.563 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-21T15:03:01.566 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,601 kB] 2024-09-21T15:03:01.568 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-21T15:03:01.570 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-21T15:03:01.571 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-21T15:03:01.571 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-21T15:03:01.577 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-21T15:03:01.577 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-21T15:03:01.577 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-21T15:03:01.578 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-21T15:03:01.593 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-21T15:03:01.594 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-21T15:03:01.594 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-21T15:03:01.595 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-21T15:03:01.621 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [818 kB] 2024-09-21T15:03:01.765 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,037 kB] 2024-09-21T15:03:01.780 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,227 kB] 2024-09-21T15:03:01.974 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [476 kB] 2024-09-21T15:03:01.976 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,227 kB] 2024-09-21T15:03:01.988 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-21T15:03:01.989 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,156 kB] 2024-09-21T15:03:02.034 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,037 kB] 2024-09-21T15:03:02.051 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [554 kB] 2024-09-21T15:03:02.059 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-21T15:03:02.060 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,283 kB] 2024-09-21T15:03:02.079 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-21T15:03:02.084 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-21T15:03:02.094 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,601 kB] 2024-09-21T15:03:02.097 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-21T15:03:02.097 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-21T15:03:02.123 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-21T15:03:02.145 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-21T15:03:02.147 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-21T15:03:02.148 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-21T15:03:02.152 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-21T15:03:02.152 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [460 kB] 2024-09-21T15:03:02.155 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-21T15:03:02.161 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-21T15:03:02.161 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,232 kB] 2024-09-21T15:03:02.197 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [476 kB] 2024-09-21T15:03:02.214 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-21T15:03:02.220 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-21T15:03:02.220 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-21T15:03:02.224 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-21T15:03:02.225 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-21T15:03:02.227 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,156 kB] 2024-09-21T15:03:02.229 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-21T15:03:02.233 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-21T15:03:02.234 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-21T15:03:02.234 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-21T15:03:02.275 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [554 kB] 2024-09-21T15:03:02.288 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-21T15:03:02.288 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,283 kB] 2024-09-21T15:03:02.296 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-21T15:03:02.296 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-21T15:03:02.296 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-21T15:03:02.297 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-21T15:03:02.321 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-21T15:03:02.335 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-21T15:03:02.335 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-21T15:03:02.358 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-21T15:03:02.371 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-21T15:03:02.377 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-21T15:03:02.377 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-21T15:03:02.378 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-21T15:03:02.379 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-21T15:03:02.380 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-21T15:03:02.380 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [460 kB] 2024-09-21T15:03:02.386 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-21T15:03:02.387 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-21T15:03:02.387 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-21T15:03:02.387 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-21T15:03:02.394 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-21T15:03:02.394 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-21T15:03:02.425 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,232 kB] 2024-09-21T15:03:02.443 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-21T15:03:02.455 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-21T15:03:02.462 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-21T15:03:02.465 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-21T15:03:02.465 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-21T15:03:02.521 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-21T15:03:02.522 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-21T15:03:02.522 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-21T15:03:02.523 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-21T15:03:02.523 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-21T15:03:02.603 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-21T15:03:02.603 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-21T15:03:02.604 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-21T15:03:05.951 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 22.0 MB in 3s (6,904 kB/s) 2024-09-21T15:03:06.317 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 22.0 MB in 4s (6,133 kB/s) 2024-09-21T15:03:06.504 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 22.0 MB in 4s (6,002 kB/s) 2024-09-21T15:03:07.613 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-21T15:03:07.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T15:03:07.718 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-21T15:03:07.869 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-21T15:03:07.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T15:03:07.939 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-09-21T15:03:07.940 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-09-21T15:03:07.971 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-21T15:03:08.090 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-21T15:03:08.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T15:03:08.151 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-09-21T15:03:08.151 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-21T15:03:08.152 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-21T15:03:08.152 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-21T15:03:08.153 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-09-21T15:03:08.153 INFO:teuthology.orchestra.run.smithi067.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-21T15:03:08.153 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-09-21T15:03:08.153 INFO:teuthology.orchestra.run.smithi067.stdout: thermald 2024-09-21T15:03:08.189 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2024-09-21T15:03:08.190 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2024-09-21T15:03:08.197 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-21T15:03:08.205 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-09-21T15:03:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-21T15:03:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-21T15:03:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-21T15:03:08.207 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-09-21T15:03:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T15:03:08.265 INFO:teuthology.orchestra.run.smithi067.stdout:3 upgraded, 5 newly installed, 0 to remove and 318 not upgraded. 2024-09-21T15:03:08.265 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 77.2 MB of archives. 2024-09-21T15:03:08.265 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-21T15:03:08.265 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-21T15:03:08.405 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2024-09-21T15:03:08.405 INFO:teuthology.orchestra.run.smithi142.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-21T15:03:08.405 INFO:teuthology.orchestra.run.smithi142.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-21T15:03:08.405 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-21T15:03:08.406 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2024-09-21T15:03:08.406 INFO:teuthology.orchestra.run.smithi142.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-21T15:03:08.406 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages: 2024-09-21T15:03:08.406 INFO:teuthology.orchestra.run.smithi142.stdout: thermald 2024-09-21T15:03:08.416 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-09-21T15:03:08.416 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-09-21T15:03:08.465 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2024-09-21T15:03:08.466 INFO:teuthology.orchestra.run.smithi142.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-21T15:03:08.466 INFO:teuthology.orchestra.run.smithi142.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-21T15:03:08.466 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-21T15:03:08.467 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded: 2024-09-21T15:03:08.468 INFO:teuthology.orchestra.run.smithi142.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T15:03:08.523 INFO:teuthology.orchestra.run.smithi142.stdout:3 upgraded, 5 newly installed, 0 to remove and 318 not upgraded. 2024-09-21T15:03:08.523 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 77.2 MB of archives. 2024-09-21T15:03:08.523 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-21T15:03:08.523 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-21T15:03:08.596 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-21T15:03:08.632 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-09-21T15:03:08.632 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-21T15:03:08.633 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-21T15:03:08.633 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-21T15:03:08.634 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-09-21T15:03:08.634 INFO:teuthology.orchestra.run.smithi146.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-21T15:03:08.634 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-09-21T15:03:08.634 INFO:teuthology.orchestra.run.smithi146.stdout: thermald 2024-09-21T15:03:08.688 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-21T15:03:08.697 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-09-21T15:03:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-21T15:03:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-21T15:03:08.697 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-21T15:03:08.699 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages will be upgraded: 2024-09-21T15:03:08.699 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T15:03:08.844 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-21T15:03:08.881 INFO:teuthology.orchestra.run.smithi146.stdout:3 upgraded, 5 newly installed, 0 to remove and 318 not upgraded. 2024-09-21T15:03:08.881 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 77.2 MB of archives. 2024-09-21T15:03:08.881 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-21T15:03:08.881 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-21T15:03:08.936 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-21T15:03:09.147 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-21T15:03:09.147 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-21T15:03:09.213 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-21T15:03:09.314 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-21T15:03:09.326 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-21T15:03:09.390 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-21T15:03:09.390 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-21T15:03:09.390 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-21T15:03:09.486 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-21T15:03:09.496 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-21T15:03:10.021 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 77.2 MB in 1s (69.5 MB/s) 2024-09-21T15:03:10.190 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 77.2 MB in 1s (75.5 MB/s) 2024-09-21T15:03:10.226 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-21T15:03:10.304 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-21T15:03:10.388 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-21T15:03:10.575 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-21T15:03:11.859 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-21T15:03:11.924 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-21T15:03:11.924 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-21T15:03:11.985 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-21T15:03:11.990 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:12.107 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-21T15:03:12.112 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:12.165 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:12.184 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:12.201 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-21T15:03:12.246 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-21T15:03:12.922 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 77.2 MB in 4s (21.8 MB/s) 2024-09-21T15:03:13.068 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-21T15:03:14.025 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-21T15:03:14.038 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:14.131 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-21T15:03:14.137 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:14.145 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:14.240 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:14.780 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-21T15:03:14.785 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:14.856 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:15.186 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-21T15:03:15.200 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:15.244 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:15.326 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-21T15:03:15.341 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:15.392 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:16.781 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-21T15:03:16.795 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:16.904 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:17.984 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-21T15:03:17.998 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:18.042 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:20.186 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:20.312 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:20.487 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:20.647 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:20.670 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:20.790 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:21.008 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:21.134 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:21.134 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-21T15:03:21.149 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-21T15:03:21.192 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-21T15:03:21.504 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-21T15:03:21.518 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-21T15:03:21.562 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-21T15:03:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:22.928 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:23.247 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:23.381 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:23.701 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-21T15:03:23.715 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-21T15:03:23.750 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-21T15:03:26.323 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-21T15:03:26.345 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:26.390 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:27.240 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-21T15:03:27.262 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:27.306 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:28.977 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:29.109 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-21T15:03:29.129 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-21T15:03:29.130 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:29.167 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:29.463 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-21T15:03:29.659 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:29.835 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-21T15:03:29.969 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:30.255 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:30.359 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:30.732 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-21T15:03:30.856 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:30.957 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-21T15:03:31.133 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:31.189 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:31.751 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-21T15:03:31.879 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-21T15:03:32.196 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-21T15:03:32.311 INFO:teuthology.orchestra.run.smithi142.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:03:32.312 INFO:teuthology.orchestra.run.smithi142.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:32.317 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:32.441 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:32.460 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-21T15:03:32.468 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:32.577 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:33.642 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-21T15:03:33.660 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:03:33.661 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:33.768 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-21T15:03:33.796 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:33.812 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:33.894 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:33.967 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T15:03:33.967 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:35.066 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-21T15:03:35.172 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-21T15:03:35.299 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:35.368 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T15:03:35.368 INFO:teuthology.orchestra.run.smithi067.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:35.447 INFO:teuthology.orchestra.run.smithi146.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:03:35.447 INFO:teuthology.orchestra.run.smithi146.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:35.623 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:36.808 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-21T15:03:36.934 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-21T15:03:37.060 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-21T15:03:37.126 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T15:03:37.126 INFO:teuthology.orchestra.run.smithi146.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:56.276 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T15:03:56.277 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub' 2024-09-21T15:03:56.291 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T15:03:56.316 INFO:teuthology.orchestra.run.smithi142.stdout:Generating grub configuration file ... 2024-09-21T15:03:56.645 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:03:56.667 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:57.038 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-21T15:03:57.049 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-21T15:03:57.774 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T15:03:57.775 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub' 2024-09-21T15:03:57.789 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T15:03:57.802 INFO:teuthology.orchestra.run.smithi067.stdout:Generating grub configuration file ... 2024-09-21T15:03:58.143 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:03:58.177 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:58.597 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-21T15:03:58.609 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-21T15:03:58.893 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T15:03:58.893 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub' 2024-09-21T15:03:58.900 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T15:03:58.911 INFO:teuthology.orchestra.run.smithi146.stdout:Generating grub configuration file ... 2024-09-21T15:03:59.189 INFO:teuthology.orchestra.run.smithi142.stdout:done 2024-09-21T15:03:59.214 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:03:59.240 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:03:59.595 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic 2024-09-21T15:03:59.639 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-21T15:03:59.651 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 20 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.4.0.196.194 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-21T15:03:59.652 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T15:03:59.653 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2024-09-21T15:03:59.653 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2024-09-21T15:03:59.653 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2024-09-21T15:03:59.653 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2024-09-21T15:03:59.653 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-21T15:03:59.653 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T15:03:59.653 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T15:03:59.653 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T15:03:59.653 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T15:03:59.653 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2024-09-21T15:03:59.748 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2024-09-21T15:03:59.883 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:03:59.883 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-21T15:03:59.889 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-21T15:03:59.907 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-21T15:04:00.754 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-09-21T15:04:01.209 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-09-21T15:04:01.254 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-09-21T15:04:01.254 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-09-21T15:04:01.254 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-09-21T15:04:01.254 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-09-21T15:04:01.254 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 20 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.4.0.196.194 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-09-21T15:04:01.255 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-09-21T15:04:01.255 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-21T15:04:01.255 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T15:04:01.255 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T15:04:01.256 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T15:04:01.256 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T15:04:01.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-09-21T15:04:01.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-09-21T15:04:01.439 INFO:teuthology.orchestra.run.smithi146.stdout:done 2024-09-21T15:04:01.498 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:04:01.499 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-21T15:04:01.506 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-21T15:04:01.524 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-21T15:04:01.751 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-21T15:04:01.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T15:04:01.846 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-21T15:04:01.854 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-09-21T15:04:01.915 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-09-21T15:04:01.915 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-09-21T15:04:01.915 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 20 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.4.0.196.194 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-09-21T15:04:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-09-21T15:04:01.916 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-21T15:04:01.916 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T15:04:01.917 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T15:04:01.917 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-21T15:04:01.917 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T15:04:01.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-09-21T15:04:02.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-09-21T15:04:02.041 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2024-09-21T15:04:02.042 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2024-09-21T15:04:02.130 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:04:02.137 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-21T15:04:02.154 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-21T15:04:02.249 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-21T15:04:02.275 INFO:teuthology.orchestra.run.smithi142.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-21T15:04:02.275 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 0 newly installed, 0 to remove and 318 not upgraded. 2024-09-21T15:04:02.277 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic 2024-09-21T15:04:02.300 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic 2024-09-21T15:04:02.300 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2024-09-21T15:04:02.300 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2024-09-21T15:04:02.300 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2024-09-21T15:04:02.300 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 20 2024-09-21T15:04:02.300 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T15:04:02.300 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.4.0.196.194 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2024-09-21T15:04:02.301 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2024-09-21T15:04:02.302 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-09-21T15:04:02.307 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.eN1o1D1KkZ 2024-09-21T15:04:02.307 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub/grub.cfg /tmp/tmp.eN1o1D1KkZ 2024-09-21T15:04:02.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.eN1o1D1KkZ 2024-09-21T15:04:02.406 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.eN1o1D1KkZ is 9KB 2024-09-21T15:04:02.420 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.eN1o1D1KkZ 2024-09-21T15:04:02.426 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T15:04:02.484 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:04:02.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T15:04:02.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T15:04:02.528 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-21T15:04:02.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo update-grub 2024-09-21T15:04:03.436 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-21T15:04:03.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T15:04:03.537 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-21T15:04:03.737 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-09-21T15:04:03.738 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-09-21T15:04:03.852 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub' 2024-09-21T15:04:03.857 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T15:04:03.873 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2024-09-21T15:04:03.982 INFO:teuthology.orchestra.run.smithi067.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-21T15:04:03.983 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 0 newly installed, 0 to remove and 318 not upgraded. 2024-09-21T15:04:03.983 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-21T15:04:03.985 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-09-21T15:04:04.007 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-09-21T15:04:04.007 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-09-21T15:04:04.007 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-09-21T15:04:04.007 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-09-21T15:04:04.007 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 20 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.4.0.196.194 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-09-21T15:04:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-09-21T15:04:04.008 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-21T15:04:04.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T15:04:04.014 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.fzWk4kZZFE 2024-09-21T15:04:04.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub/grub.cfg /tmp/tmp.fzWk4kZZFE 2024-09-21T15:04:04.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.fzWk4kZZFE 2024-09-21T15:04:04.080 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-21T15:04:04.112 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.fzWk4kZZFE is 9KB 2024-09-21T15:04:04.119 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:04:04.126 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.fzWk4kZZFE 2024-09-21T15:04:04.131 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:04:04.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T15:04:04.190 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:04:04.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T15:04:04.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T15:04:04.230 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-21T15:04:04.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo update-grub 2024-09-21T15:04:04.299 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-09-21T15:04:04.299 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-09-21T15:04:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-21T15:04:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-21T15:04:04.538 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-21T15:04:04.538 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 0 newly installed, 0 to remove and 318 not upgraded. 2024-09-21T15:04:04.540 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-09-21T15:04:04.561 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-09-21T15:04:04.561 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 20 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.4.0.196.194 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-09-21T15:04:04.562 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-09-21T15:04:04.563 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-09-21T15:04:04.568 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.vUZqkcDEpH 2024-09-21T15:04:04.569 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vUZqkcDEpH 2024-09-21T15:04:04.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.vUZqkcDEpH 2024-09-21T15:04:04.618 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.vUZqkcDEpH is 9KB 2024-09-21T15:04:04.632 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.vUZqkcDEpH 2024-09-21T15:04:04.638 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T15:04:04.697 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:04:04.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T15:04:04.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T15:04:04.735 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-21T15:04:04.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo update-grub 2024-09-21T15:04:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:done 2024-09-21T15:04:05.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2024-09-21T15:04:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2024-09-21T15:04:05.647 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T15:04:05.659 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-09-21T15:04:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:04:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:04:06.053 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub' 2024-09-21T15:04:06.062 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T15:04:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-09-21T15:04:06.343 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-21T15:04:06.355 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-21T15:04:06.367 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-21T15:04:06.379 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-21T15:04:06.725 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-21T15:04:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-21T15:04:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-09-21T15:04:07.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-09-21T15:04:07.496 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-09-21T15:04:07.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-09-21T15:04:35.541 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T15:04:35.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-21T15:04:35.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:04:37.445 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T15:04:37.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-21T15:04:37.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:04:37.501 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-21T15:04:37.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-21T15:04:37.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:05:10.127 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-09-21T15:05:11.919 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-21T15:05:12.175 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-21T15:05:19.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-21T15:05:19.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:05:20.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-21T15:05:20.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:05:21.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-21T15:05:21.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:05:22.191 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-09-21T15:05:34.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-21T15:05:34.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:05:39.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-09-21T15:05:51.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-21T15:05:51.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:05:51.765 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-21T15:05:52.623 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-21T15:05:52.846 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-21T15:05:52.846 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-21T15:05:52.846 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-09-21T15:05:52.894 INFO:teuthology.orchestra.run.smithi067.stdout:5.4.0-196-generic 2024-09-21T15:05:52.895 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-21T15:05:52.895 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T15:05:52.895 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T15:05:53.896 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T15:05:53.896 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T15:05:54.009 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-09-21T15:05:54.035 DEBUG:teuthology.parallel:result is None 2024-09-21T15:06:04.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-21T15:06:04.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:06:04.859 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-21T15:06:05.407 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-21T15:06:05.407 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.4.0-196-generic"... 2024-09-21T15:06:05.407 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-21T15:06:05.413 INFO:teuthology.orchestra.run.smithi146.stdout:5.4.0-196-generic 2024-09-21T15:06:05.414 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-21T15:06:05.414 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T15:06:05.414 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T15:06:06.414 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-09-21T15:06:06.414 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T15:06:06.799 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-09-21T15:06:06.817 DEBUG:teuthology.parallel:result is None 2024-09-21T15:06:34.199 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T15:06:49.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-21T15:06:49.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:06:49.438 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-09-21T15:06:49.986 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-21T15:06:49.986 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-196-generic"... 2024-09-21T15:06:49.987 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-09-21T15:06:49.993 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-196-generic 2024-09-21T15:06:49.993 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-21T15:06:49.993 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T15:06:49.993 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T15:06:50.994 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T15:06:50.994 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T15:06:51.098 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2024-09-21T15:06:51.118 DEBUG:teuthology.parallel:result is None 2024-09-21T15:06:51.118 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T15:06:51.126 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T15:06:51.126 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T15:06:51.128 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T15:06:51.130 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T15:06:51.137 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T15:06:51.148 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T15:06:51.158 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T15:06:51.158 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T15:06:51.178 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T15:06:51.180 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T15:06:51.277 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T15:06:51.283 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T15:06:51.283 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T15:06:51.285 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T15:06:51.287 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T15:06:51.307 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:06:51.308 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:06:51.318 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:06:51.320 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:06:51.323 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:06:51.334 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:06:51.336 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T15:06:51.343 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T15:06:51.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T15:06:51.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T15:06:51.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T15:06:51.385 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T15:06:51.393 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T15:06:51.393 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T15:06:51.395 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T15:06:51.397 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T15:06:51.432 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T15:06:51.439 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T15:06:51.487 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:06:51.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T15:06:51.517 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T15:06:51.524 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T15:06:51.574 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:06:51.575 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T15:06:51.598 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T15:06:51.605 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T15:06:51.612 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:06:51.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T15:06:51.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-09-21T15:06:51.641 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2024-09-21T15:06:51.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-09-21T15:06:51.732 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T15:06:51.745 INFO:teuthology.task.internal:Starting timer... 2024-09-21T15:06:51.745 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T15:06:51.778 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T15:06:51.791 DEBUG:teuthology.task.selinux:Excluding smithi067: OS 'ubuntu' does not support SELinux 2024-09-21T15:06:51.791 DEBUG:teuthology.task.selinux:Excluding smithi142: OS 'ubuntu' does not support SELinux 2024-09-21T15:06:51.792 DEBUG:teuthology.task.selinux:Excluding smithi146: OS 'ubuntu' does not support SELinux 2024-09-21T15:06:51.792 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T15:06:51.792 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-21T15:06:51.792 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T15:06:51.792 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T15:06:51.801 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T15:06:51.811 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T15:06:51.812 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi142.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T15:12:28.894 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-09-21T15:12:28.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-21T15:12:28.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:12:28.995 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-21T15:12:29.120 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-21T15:12:29.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-21T15:12:29.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:12:29.208 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-09-21T15:12:29.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-21T15:12:29.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-21T15:12:29.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:12:29.418 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-21T15:12:29.540 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-21T15:12:29.541 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T15:12:29.550 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T15:12:29.551 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T15:12:29.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:12:29.554 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T15:12:29.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:12:29.556 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T15:12:29.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:12:29.588 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-21T15:12:29.588 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Command line: ntpd -gq 2024-09-21T15:12:29.590 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: proto: precision = 0.092 usec (-23) 2024-09-21T15:12:29.590 INFO:teuthology.orchestra.run.smithi067.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T15:12:29.590 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T15:12:29.591 INFO:teuthology.orchestra.run.smithi067.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Listen normally on 3 enp3s0f1 172.21.15.67:123 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Listen normally on 4 lo [::1]:123 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:fc0b%5]:123 2024-09-21T15:12:29.592 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:29 ntpd[13702]: Listening on routing socket on fd #22 for interface updates 2024-09-21T15:12:29.593 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-21T15:12:29.593 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Command line: ntpd -gq 2024-09-21T15:12:29.593 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: proto: precision = 0.105 usec (-23) 2024-09-21T15:12:29.593 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-21T15:12:29.593 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Command line: ntpd -gq 2024-09-21T15:12:29.593 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: proto: precision = 0.092 usec (-23) 2024-09-21T15:12:29.594 INFO:teuthology.orchestra.run.smithi146.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T15:12:29.594 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T15:12:29.594 INFO:teuthology.orchestra.run.smithi146.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T15:12:29.595 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T15:12:29.595 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T15:12:29.595 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T15:12:29.595 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T15:12:29.596 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Listen normally on 3 enp3s0f1 172.21.15.146:123 2024-09-21T15:12:29.596 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Listen normally on 4 lo [::1]:123 2024-09-21T15:12:29.596 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:76fd%5]:123 2024-09-21T15:12:29.596 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:29 ntpd[13782]: Listening on routing socket on fd #22 for interface updates 2024-09-21T15:12:29.596 INFO:teuthology.orchestra.run.smithi142.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T15:12:29.596 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T15:12:29.597 INFO:teuthology.orchestra.run.smithi142.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T15:12:29.597 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T15:12:29.598 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T15:12:29.598 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T15:12:29.598 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T15:12:29.598 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Listen normally on 3 enp3s0f1 172.21.15.142:123 2024-09-21T15:12:29.598 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Listen normally on 4 lo [::1]:123 2024-09-21T15:12:29.598 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e0d%5]:123 2024-09-21T15:12:29.598 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:29 ntpd[13759]: Listening on routing socket on fd #22 for interface updates 2024-09-21T15:12:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:30 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.595 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:30 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:30.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:30.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:30.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:30.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:30.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:30 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:31.591 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:31 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:31.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:31 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:32.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:32 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:32.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:32.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:32.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:32.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:32 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:32.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:32.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:32 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:33 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:33 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:33 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:33.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:33 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:33.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:33 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:33.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:33 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:34 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:34 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:34 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:35 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:35 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:35 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:35 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:35 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:35 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:36.589 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stdout:21 Sep 15:12:36 ntpd[13702]: ntpd: time slew +0.002682 s 2024-09-21T15:12:36.590 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew +0.002682s 2024-09-21T15:12:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T15:12:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:21 Sep 15:12:36 ntpd[13702]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T15:12:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stdout:21 Sep 15:12:36 ntpd[13782]: ntpd: time slew +0.000542 s 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stdout:ntpd: time slew +0.000542s 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T15:12:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:21 Sep 15:12:36 ntpd[13782]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T15:12:36.595 INFO:teuthology.orchestra.run.smithi142.stdout:21 Sep 15:12:36 ntpd[13759]: ntpd: time slew -0.000080 s 2024-09-21T15:12:36.596 INFO:teuthology.orchestra.run.smithi142.stdout:ntpd: time slew -0.000080s 2024-09-21T15:12:36.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T15:12:36.596 INFO:teuthology.orchestra.run.smithi142.stderr:21 Sep 15:12:36 ntpd[13759]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T15:12:36.654 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T15:12:36.654 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-09-21T15:12:36.654 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.654 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.654 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.654 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T15:12:36.655 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-09-21T15:12:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.666 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T15:12:36.666 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2024-09-21T15:12:36.666 INFO:teuthology.orchestra.run.smithi142.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.666 INFO:teuthology.orchestra.run.smithi142.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.666 INFO:teuthology.orchestra.run.smithi142.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.666 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:12:36.667 INFO:teuthology.run_tasks:Running task install... 2024-09-21T15:12:36.676 DEBUG:teuthology.task.install:project ceph 2024-09-21T15:12:36.677 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}} 2024-09-21T15:12:36.677 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'} 2024-09-21T15:12:36.677 INFO:teuthology.task.install:Using flavor: default 2024-09-21T15:12:36.682 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T15:12:36.682 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-09-21T15:12:36.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-key list | grep Ceph 2024-09-21T15:12:36.683 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-key list | grep Ceph 2024-09-21T15:12:36.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-key list | grep Ceph 2024-09-21T15:12:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-21T15:12:36.789 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-21T15:12:36.794 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-21T15:12:36.842 INFO:teuthology.orchestra.run.smithi142.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T15:12:36.842 INFO:teuthology.orchestra.run.smithi142.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T15:12:36.843 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-21T15:12:36.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:12:36.852 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T15:12:36.852 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T15:12:36.853 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-21T15:12:36.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:12:36.859 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T15:12:36.859 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T15:12:36.859 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-21T15:12:36.859 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:12:36.987 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default/ 2024-09-21T15:12:36.987 INFO:teuthology.task.install.deb:Package version is 18.2.4-855-g19e2a2be-1focal 2024-09-21T15:12:36.998 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default/ 2024-09-21T15:12:36.998 INFO:teuthology.task.install.deb:Package version is 18.2.4-855-g19e2a2be-1focal 2024-09-21T15:12:37.004 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default/ 2024-09-21T15:12:37.004 INFO:teuthology.task.install.deb:Package version is 18.2.4-855-g19e2a2be-1focal 2024-09-21T15:12:37.096 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:12:37.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T15:12:37.108 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:12:37.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T15:12:37.112 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:12:37.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T15:12:37.113 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2024-09-21T15:12:37.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-09-21T15:12:37.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-09-21T15:12:37.250 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-21T15:12:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:12:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-21T15:12:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-21T15:12:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-21T15:12:37.307 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal InRelease 2024-09-21T15:12:37.316 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:12:37.316 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-21T15:12:37.321 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-21T15:12:37.322 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal InRelease 2024-09-21T15:12:37.329 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-21T15:12:37.339 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-21T15:12:37.344 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-21T15:12:37.352 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release.gpg 2024-09-21T15:12:37.367 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release.gpg 2024-09-21T15:12:37.369 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal InRelease 2024-09-21T15:12:37.370 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-21T15:12:37.373 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main i386 Packages [3,426 B] 2024-09-21T15:12:37.389 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-21T15:12:37.390 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-21T15:12:37.410 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-21T15:12:37.414 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal Release.gpg 2024-09-21T15:12:37.421 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main i386 Packages [3,426 B] 2024-09-21T15:12:37.435 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main i386 Packages [3,426 B] 2024-09-21T15:12:37.452 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-21T15:12:37.468 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-21T15:12:37.536 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-21T15:12:37.908 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 28.7 kB in 1s (45.4 kB/s) 2024-09-21T15:12:37.936 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 28.7 kB in 1s (44.4 kB/s) 2024-09-21T15:12:37.974 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 28.7 kB in 1s (45.2 kB/s) 2024-09-21T15:12:39.068 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-21T15:12:39.093 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-855-g19e2a2be-1focal cephadm=18.2.4-855-g19e2a2be-1focal ceph-mds=18.2.4-855-g19e2a2be-1focal ceph-mgr=18.2.4-855-g19e2a2be-1focal ceph-common=18.2.4-855-g19e2a2be-1focal ceph-fuse=18.2.4-855-g19e2a2be-1focal ceph-test=18.2.4-855-g19e2a2be-1focal ceph-volume=18.2.4-855-g19e2a2be-1focal radosgw=18.2.4-855-g19e2a2be-1focal python3-rados=18.2.4-855-g19e2a2be-1focal python3-rgw=18.2.4-855-g19e2a2be-1focal python3-cephfs=18.2.4-855-g19e2a2be-1focal python3-rbd=18.2.4-855-g19e2a2be-1focal libcephfs2=18.2.4-855-g19e2a2be-1focal libcephfs-dev=18.2.4-855-g19e2a2be-1focal librados2=18.2.4-855-g19e2a2be-1focal librbd1=18.2.4-855-g19e2a2be-1focal rbd-fuse=18.2.4-855-g19e2a2be-1focal rbd-nbd=18.2.4-855-g19e2a2be-1focal 2024-09-21T15:12:39.138 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-21T15:12:39.140 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-21T15:12:39.163 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-21T15:12:39.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-855-g19e2a2be-1focal cephadm=18.2.4-855-g19e2a2be-1focal ceph-mds=18.2.4-855-g19e2a2be-1focal ceph-mgr=18.2.4-855-g19e2a2be-1focal ceph-common=18.2.4-855-g19e2a2be-1focal ceph-fuse=18.2.4-855-g19e2a2be-1focal ceph-test=18.2.4-855-g19e2a2be-1focal ceph-volume=18.2.4-855-g19e2a2be-1focal radosgw=18.2.4-855-g19e2a2be-1focal python3-rados=18.2.4-855-g19e2a2be-1focal python3-rgw=18.2.4-855-g19e2a2be-1focal python3-cephfs=18.2.4-855-g19e2a2be-1focal python3-rbd=18.2.4-855-g19e2a2be-1focal libcephfs2=18.2.4-855-g19e2a2be-1focal libcephfs-dev=18.2.4-855-g19e2a2be-1focal librados2=18.2.4-855-g19e2a2be-1focal librbd1=18.2.4-855-g19e2a2be-1focal rbd-fuse=18.2.4-855-g19e2a2be-1focal rbd-nbd=18.2.4-855-g19e2a2be-1focal 2024-09-21T15:12:39.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-855-g19e2a2be-1focal cephadm=18.2.4-855-g19e2a2be-1focal ceph-mds=18.2.4-855-g19e2a2be-1focal ceph-mgr=18.2.4-855-g19e2a2be-1focal ceph-common=18.2.4-855-g19e2a2be-1focal ceph-fuse=18.2.4-855-g19e2a2be-1focal ceph-test=18.2.4-855-g19e2a2be-1focal ceph-volume=18.2.4-855-g19e2a2be-1focal radosgw=18.2.4-855-g19e2a2be-1focal python3-rados=18.2.4-855-g19e2a2be-1focal python3-rgw=18.2.4-855-g19e2a2be-1focal python3-cephfs=18.2.4-855-g19e2a2be-1focal python3-rbd=18.2.4-855-g19e2a2be-1focal libcephfs2=18.2.4-855-g19e2a2be-1focal libcephfs-dev=18.2.4-855-g19e2a2be-1focal librados2=18.2.4-855-g19e2a2be-1focal librbd1=18.2.4-855-g19e2a2be-1focal rbd-fuse=18.2.4-855-g19e2a2be-1focal rbd-nbd=18.2.4-855-g19e2a2be-1focal 2024-09-21T15:12:39.235 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-21T15:12:39.242 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-09-21T15:12:39.357 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2024-09-21T15:12:39.358 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2024-09-21T15:12:39.444 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-09-21T15:12:39.518 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-09-21T15:12:39.519 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-09-21T15:12:39.519 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-09-21T15:12:39.550 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T15:12:39.551 INFO:teuthology.orchestra.run.smithi142.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-21T15:12:39.551 INFO:teuthology.orchestra.run.smithi142.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T15:12:39.551 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2024-09-21T15:12:39.551 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T15:12:39.551 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-21T15:12:39.551 INFO:teuthology.orchestra.run.smithi142.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-21T15:12:39.551 INFO:teuthology.orchestra.run.smithi142.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-21T15:12:39.552 INFO:teuthology.orchestra.run.smithi142.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-21T15:12:39.552 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-21T15:12:39.552 INFO:teuthology.orchestra.run.smithi142.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-21T15:12:39.552 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-21T15:12:39.552 INFO:teuthology.orchestra.run.smithi142.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-21T15:12:39.553 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-21T15:12:39.554 INFO:teuthology.orchestra.run.smithi142.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-21T15:12:39.555 INFO:teuthology.orchestra.run.smithi142.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T15:12:39.555 INFO:teuthology.orchestra.run.smithi142.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-21T15:12:39.555 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages: 2024-09-21T15:12:39.555 INFO:teuthology.orchestra.run.smithi142.stdout: btrfs-tools 2024-09-21T15:12:39.650 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T15:12:39.651 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-21T15:12:39.652 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T15:12:39.652 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-09-21T15:12:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T15:12:39.652 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-21T15:12:39.652 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-21T15:12:39.654 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-21T15:12:39.655 INFO:teuthology.orchestra.run.smithi067.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-21T15:12:39.656 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-09-21T15:12:39.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-21T15:12:39.656 INFO:teuthology.orchestra.run.smithi067.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-21T15:12:39.656 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-09-21T15:12:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: btrfs-tools 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-21T15:12:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-21T15:12:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-09-21T15:12:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: btrfs-tools 2024-09-21T15:12:39.671 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2024-09-21T15:12:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T15:12:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T15:12:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-21T15:12:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-21T15:12:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-21T15:12:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T15:12:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-21T15:12:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-21T15:12:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-21T15:12:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-21T15:12:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-21T15:12:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse 2024-09-21T15:12:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd socat xmlstarlet 2024-09-21T15:12:39.674 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded: 2024-09-21T15:12:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 librbd1 2024-09-21T15:12:39.741 INFO:teuthology.orchestra.run.smithi142.stdout:2 upgraded, 111 newly installed, 0 to remove and 297 not upgraded. 2024-09-21T15:12:39.741 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 199 MB of archives. 2024-09-21T15:12:39.741 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 978 MB of additional disk space will be used. 2024-09-21T15:12:39.741 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-21T15:12:39.777 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-09-21T15:12:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T15:12:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T15:12:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-21T15:12:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-21T15:12:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-21T15:12:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T15:12:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-21T15:12:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-21T15:12:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-21T15:12:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-21T15:12:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-21T15:12:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-21T15:12:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-21T15:12:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd socat xmlstarlet 2024-09-21T15:12:39.781 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-09-21T15:12:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 librbd1 2024-09-21T15:12:39.783 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-21T15:12:39.784 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd socat xmlstarlet 2024-09-21T15:12:39.785 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages will be upgraded: 2024-09-21T15:12:39.786 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 librbd1 2024-09-21T15:12:39.825 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-855-g19e2a2be-1focal [3,453 kB] 2024-09-21T15:12:39.826 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-21T15:12:39.829 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-21T15:12:39.846 INFO:teuthology.orchestra.run.smithi067.stdout:2 upgraded, 111 newly installed, 0 to remove and 297 not upgraded. 2024-09-21T15:12:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 199 MB of archives. 2024-09-21T15:12:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 978 MB of additional disk space will be used. 2024-09-21T15:12:39.847 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-21T15:12:39.848 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-21T15:12:39.902 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-21T15:12:39.904 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-21T15:12:39.912 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-21T15:12:39.913 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-21T15:12:39.914 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-21T15:12:39.915 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-21T15:12:39.915 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-21T15:12:39.918 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-21T15:12:39.918 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-21T15:12:39.920 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-21T15:12:39.930 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-855-g19e2a2be-1focal [3,453 kB] 2024-09-21T15:12:39.930 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-21T15:12:39.931 INFO:teuthology.orchestra.run.smithi146.stdout:2 upgraded, 111 newly installed, 0 to remove and 297 not upgraded. 2024-09-21T15:12:39.931 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 199 MB of archives. 2024-09-21T15:12:39.931 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 978 MB of additional disk space will be used. 2024-09-21T15:12:39.932 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-855-g19e2a2be-1focal [3,453 kB] 2024-09-21T15:12:39.933 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-21T15:12:39.937 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-21T15:12:39.937 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-21T15:12:39.938 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-21T15:12:39.940 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T15:12:39.940 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-21T15:12:39.940 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-21T15:12:39.941 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-21T15:12:39.942 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-21T15:12:39.943 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-21T15:12:39.951 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-21T15:12:39.954 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-21T15:12:39.963 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-21T15:12:39.970 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-21T15:12:39.971 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-21T15:12:39.971 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-21T15:12:39.972 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-21T15:12:39.974 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-21T15:12:39.977 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-21T15:12:39.978 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-21T15:12:39.978 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-21T15:12:39.980 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-21T15:12:39.989 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-21T15:12:39.991 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-21T15:12:39.992 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-21T15:12:39.993 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-855-g19e2a2be-1focal [4,044 kB] 2024-09-21T15:12:40.007 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-21T15:12:40.008 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-21T15:12:40.008 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-21T15:12:40.009 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-21T15:12:40.010 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-21T15:12:40.011 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-21T15:12:40.011 INFO:teuthology.orchestra.run.smithi142.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-21T15:12:40.012 INFO:teuthology.orchestra.run.smithi142.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-21T15:12:40.013 INFO:teuthology.orchestra.run.smithi142.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-21T15:12:40.016 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-21T15:12:40.018 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-21T15:12:40.019 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-21T15:12:40.020 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-21T15:12:40.020 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-21T15:12:40.024 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-21T15:12:40.024 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-21T15:12:40.025 INFO:teuthology.orchestra.run.smithi142.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T15:12:40.026 INFO:teuthology.orchestra.run.smithi142.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-21T15:12:40.027 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-21T15:12:40.028 INFO:teuthology.orchestra.run.smithi142.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-21T15:12:40.043 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-21T15:12:40.043 INFO:teuthology.orchestra.run.smithi142.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-21T15:12:40.044 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-21T15:12:40.044 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-21T15:12:40.045 INFO:teuthology.orchestra.run.smithi142.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-21T15:12:40.045 INFO:teuthology.orchestra.run.smithi142.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-21T15:12:40.046 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T15:12:40.046 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-21T15:12:40.046 INFO:teuthology.orchestra.run.smithi142.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-21T15:12:40.046 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-21T15:12:40.048 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-21T15:12:40.048 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-21T15:12:40.050 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-21T15:12:40.056 INFO:teuthology.orchestra.run.smithi142.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-855-g19e2a2be-1focal [746 kB] 2024-09-21T15:12:40.067 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-21T15:12:40.067 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-21T15:12:40.068 INFO:teuthology.orchestra.run.smithi142.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-855-g19e2a2be-1focal [388 kB] 2024-09-21T15:12:40.074 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-21T15:12:40.078 INFO:teuthology.orchestra.run.smithi142.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-855-g19e2a2be-1focal [31.9 kB] 2024-09-21T15:12:40.079 INFO:teuthology.orchestra.run.smithi142.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-855-g19e2a2be-1focal [201 kB] 2024-09-21T15:12:40.081 INFO:teuthology.orchestra.run.smithi142.stdout:Get:57 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-855-g19e2a2be-1focal [62.5 kB] 2024-09-21T15:12:40.082 INFO:teuthology.orchestra.run.smithi142.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-855-g19e2a2be-1focal [367 kB] 2024-09-21T15:12:40.089 INFO:teuthology.orchestra.run.smithi142.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-855-g19e2a2be-1focal [6,627 kB] 2024-09-21T15:12:40.091 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-21T15:12:40.091 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-21T15:12:40.092 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-21T15:12:40.094 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-21T15:12:40.098 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-21T15:12:40.099 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-21T15:12:40.103 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-21T15:12:40.116 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-21T15:12:40.118 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-21T15:12:40.119 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-21T15:12:40.138 INFO:teuthology.orchestra.run.smithi142.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-21T15:12:40.140 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-21T15:12:40.140 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-21T15:12:40.141 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-21T15:12:40.142 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-21T15:12:40.143 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-21T15:12:40.144 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-21T15:12:40.144 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-21T15:12:40.158 INFO:teuthology.orchestra.run.smithi142.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-21T15:12:40.159 INFO:teuthology.orchestra.run.smithi142.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-21T15:12:40.163 INFO:teuthology.orchestra.run.smithi067.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T15:12:40.163 INFO:teuthology.orchestra.run.smithi067.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-21T15:12:40.166 INFO:teuthology.orchestra.run.smithi067.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-21T15:12:40.173 INFO:teuthology.orchestra.run.smithi142.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-21T15:12:40.173 INFO:teuthology.orchestra.run.smithi142.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-21T15:12:40.182 INFO:teuthology.orchestra.run.smithi067.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-21T15:12:40.184 INFO:teuthology.orchestra.run.smithi067.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-21T15:12:40.185 INFO:teuthology.orchestra.run.smithi067.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-21T15:12:40.186 INFO:teuthology.orchestra.run.smithi067.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-21T15:12:40.230 INFO:teuthology.orchestra.run.smithi142.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-21T15:12:40.230 INFO:teuthology.orchestra.run.smithi142.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-855-g19e2a2be-1focal [113 kB] 2024-09-21T15:12:40.231 INFO:teuthology.orchestra.run.smithi142.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-21T15:12:40.271 INFO:teuthology.orchestra.run.smithi067.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-21T15:12:40.281 INFO:teuthology.orchestra.run.smithi142.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-21T15:12:40.281 INFO:teuthology.orchestra.run.smithi142.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-855-g19e2a2be-1focal [478 kB] 2024-09-21T15:12:40.282 INFO:teuthology.orchestra.run.smithi142.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-21T15:12:40.283 INFO:teuthology.orchestra.run.smithi142.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-21T15:12:40.285 INFO:teuthology.orchestra.run.smithi142.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-21T15:12:40.285 INFO:teuthology.orchestra.run.smithi142.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-21T15:12:40.286 INFO:teuthology.orchestra.run.smithi142.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-21T15:12:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-21T15:12:40.290 INFO:teuthology.orchestra.run.smithi142.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-21T15:12:40.290 INFO:teuthology.orchestra.run.smithi067.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-21T15:12:40.292 INFO:teuthology.orchestra.run.smithi142.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-21T15:12:40.293 INFO:teuthology.orchestra.run.smithi142.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-855-g19e2a2be-1focal [25.2 MB] 2024-09-21T15:12:40.303 INFO:teuthology.orchestra.run.smithi142.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-21T15:12:40.305 INFO:teuthology.orchestra.run.smithi067.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-21T15:12:40.305 INFO:teuthology.orchestra.run.smithi067.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-21T15:12:40.305 INFO:teuthology.orchestra.run.smithi067.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-21T15:12:40.316 INFO:teuthology.orchestra.run.smithi142.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-21T15:12:40.320 INFO:teuthology.orchestra.run.smithi142.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-21T15:12:40.324 INFO:teuthology.orchestra.run.smithi142.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-21T15:12:40.343 INFO:teuthology.orchestra.run.smithi142.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-21T15:12:40.345 INFO:teuthology.orchestra.run.smithi142.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-21T15:12:40.347 INFO:teuthology.orchestra.run.smithi142.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-21T15:12:40.359 INFO:teuthology.orchestra.run.smithi067.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-21T15:12:40.360 INFO:teuthology.orchestra.run.smithi067.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-21T15:12:40.361 INFO:teuthology.orchestra.run.smithi067.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-21T15:12:40.361 INFO:teuthology.orchestra.run.smithi067.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-21T15:12:40.362 INFO:teuthology.orchestra.run.smithi067.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-21T15:12:40.363 INFO:teuthology.orchestra.run.smithi067.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-21T15:12:40.363 INFO:teuthology.orchestra.run.smithi067.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-21T15:12:40.366 INFO:teuthology.orchestra.run.smithi067.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-21T15:12:40.378 INFO:teuthology.orchestra.run.smithi067.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-21T15:12:40.388 INFO:teuthology.orchestra.run.smithi067.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-21T15:12:40.388 INFO:teuthology.orchestra.run.smithi142.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-21T15:12:40.389 INFO:teuthology.orchestra.run.smithi067.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-21T15:12:40.389 INFO:teuthology.orchestra.run.smithi142.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-21T15:12:40.389 INFO:teuthology.orchestra.run.smithi142.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-21T15:12:40.390 INFO:teuthology.orchestra.run.smithi142.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-21T15:12:40.393 INFO:teuthology.orchestra.run.smithi067.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-21T15:12:40.393 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-21T15:12:40.397 INFO:teuthology.orchestra.run.smithi067.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-21T15:12:40.403 INFO:teuthology.orchestra.run.smithi142.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T15:12:40.404 INFO:teuthology.orchestra.run.smithi142.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-21T15:12:40.405 INFO:teuthology.orchestra.run.smithi142.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-21T15:12:40.405 INFO:teuthology.orchestra.run.smithi142.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-21T15:12:40.406 INFO:teuthology.orchestra.run.smithi142.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-21T15:12:40.407 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-21T15:12:40.415 INFO:teuthology.orchestra.run.smithi067.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-21T15:12:40.417 INFO:teuthology.orchestra.run.smithi067.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-21T15:12:40.419 INFO:teuthology.orchestra.run.smithi067.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-21T15:12:40.457 INFO:teuthology.orchestra.run.smithi067.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-21T15:12:40.457 INFO:teuthology.orchestra.run.smithi067.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-21T15:12:40.458 INFO:teuthology.orchestra.run.smithi067.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-21T15:12:40.458 INFO:teuthology.orchestra.run.smithi067.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-21T15:12:40.469 INFO:teuthology.orchestra.run.smithi067.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T15:12:40.470 INFO:teuthology.orchestra.run.smithi067.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-21T15:12:40.471 INFO:teuthology.orchestra.run.smithi067.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-21T15:12:40.471 INFO:teuthology.orchestra.run.smithi067.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-21T15:12:40.472 INFO:teuthology.orchestra.run.smithi067.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-21T15:12:40.495 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-21T15:12:40.741 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-21T15:12:40.750 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-21T15:12:40.778 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-21T15:12:40.787 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-21T15:12:40.787 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-21T15:12:40.795 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-21T15:12:40.796 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-21T15:12:40.805 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-21T15:12:40.807 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-21T15:12:40.823 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-21T15:12:40.832 INFO:teuthology.orchestra.run.smithi142.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-855-g19e2a2be-1focal [9,605 kB] 2024-09-21T15:12:40.845 INFO:teuthology.orchestra.run.smithi067.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-855-g19e2a2be-1focal [4,044 kB] 2024-09-21T15:12:40.895 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-21T15:12:40.896 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-21T15:12:40.896 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-21T15:12:40.899 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T15:12:40.900 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-855-g19e2a2be-1focal [4,044 kB] 2024-09-21T15:12:40.900 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-21T15:12:40.900 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-21T15:12:40.903 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-21T15:12:40.904 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-21T15:12:40.906 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-21T15:12:40.956 INFO:teuthology.orchestra.run.smithi142.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-855-g19e2a2be-1focal [238 kB] 2024-09-21T15:12:40.961 INFO:teuthology.orchestra.run.smithi142.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-855-g19e2a2be-1focal [148 kB] 2024-09-21T15:12:40.971 INFO:teuthology.orchestra.run.smithi142.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-855-g19e2a2be-1focal [1,574 kB] 2024-09-21T15:12:40.986 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-21T15:12:40.986 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-21T15:12:40.996 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-21T15:12:40.997 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-21T15:12:40.997 INFO:teuthology.orchestra.run.smithi142.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-855-g19e2a2be-1focal [6,640 kB] 2024-09-21T15:12:40.998 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-21T15:12:40.999 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-21T15:12:41.001 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-21T15:12:41.003 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-21T15:12:41.067 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-21T15:12:41.073 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-21T15:12:41.127 INFO:teuthology.orchestra.run.smithi142.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-855-g19e2a2be-1focal [24.5 MB] 2024-09-21T15:12:41.149 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-21T15:12:41.152 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-21T15:12:41.155 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-21T15:12:41.156 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-21T15:12:41.157 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-21T15:12:41.158 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-21T15:12:41.160 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-21T15:12:41.161 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-21T15:12:41.230 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-21T15:12:41.231 INFO:teuthology.orchestra.run.smithi146.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-21T15:12:41.244 INFO:teuthology.orchestra.run.smithi146.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T15:12:41.244 INFO:teuthology.orchestra.run.smithi146.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-21T15:12:41.247 INFO:teuthology.orchestra.run.smithi146.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-21T15:12:41.249 INFO:teuthology.orchestra.run.smithi146.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-21T15:12:41.313 INFO:teuthology.orchestra.run.smithi146.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-21T15:12:41.314 INFO:teuthology.orchestra.run.smithi146.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-21T15:12:41.315 INFO:teuthology.orchestra.run.smithi146.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-21T15:12:41.467 INFO:teuthology.orchestra.run.smithi067.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-855-g19e2a2be-1focal [746 kB] 2024-09-21T15:12:41.593 INFO:teuthology.orchestra.run.smithi067.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-855-g19e2a2be-1focal [388 kB] 2024-09-21T15:12:41.596 INFO:teuthology.orchestra.run.smithi146.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-21T15:12:41.647 INFO:teuthology.orchestra.run.smithi146.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-855-g19e2a2be-1focal [746 kB] 2024-09-21T15:12:41.655 INFO:teuthology.orchestra.run.smithi067.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-855-g19e2a2be-1focal [31.9 kB] 2024-09-21T15:12:41.657 INFO:teuthology.orchestra.run.smithi067.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-855-g19e2a2be-1focal [201 kB] 2024-09-21T15:12:41.661 INFO:teuthology.orchestra.run.smithi067.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-855-g19e2a2be-1focal [62.5 kB] 2024-09-21T15:12:41.663 INFO:teuthology.orchestra.run.smithi067.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-855-g19e2a2be-1focal [367 kB] 2024-09-21T15:12:41.670 INFO:teuthology.orchestra.run.smithi146.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-21T15:12:41.673 INFO:teuthology.orchestra.run.smithi146.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-21T15:12:41.704 INFO:teuthology.orchestra.run.smithi146.stdout:Get:57 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-855-g19e2a2be-1focal [388 kB] 2024-09-21T15:12:41.710 INFO:teuthology.orchestra.run.smithi067.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-855-g19e2a2be-1focal [6,627 kB] 2024-09-21T15:12:41.720 INFO:teuthology.orchestra.run.smithi142.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-855-g19e2a2be-1focal [13.7 kB] 2024-09-21T15:12:41.720 INFO:teuthology.orchestra.run.smithi142.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-855-g19e2a2be-1focal [856 kB] 2024-09-21T15:12:41.732 INFO:teuthology.orchestra.run.smithi146.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-855-g19e2a2be-1focal [31.9 kB] 2024-09-21T15:12:41.732 INFO:teuthology.orchestra.run.smithi142.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-855-g19e2a2be-1focal [2,295 kB] 2024-09-21T15:12:41.733 INFO:teuthology.orchestra.run.smithi146.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-855-g19e2a2be-1focal [201 kB] 2024-09-21T15:12:41.736 INFO:teuthology.orchestra.run.smithi146.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-855-g19e2a2be-1focal [62.5 kB] 2024-09-21T15:12:41.742 INFO:teuthology.orchestra.run.smithi146.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-21T15:12:41.742 INFO:teuthology.orchestra.run.smithi146.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-21T15:12:41.743 INFO:teuthology.orchestra.run.smithi146.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-21T15:12:41.746 INFO:teuthology.orchestra.run.smithi146.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-21T15:12:41.746 INFO:teuthology.orchestra.run.smithi146.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-21T15:12:41.746 INFO:teuthology.orchestra.run.smithi146.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-21T15:12:41.746 INFO:teuthology.orchestra.run.smithi146.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-21T15:12:41.749 INFO:teuthology.orchestra.run.smithi142.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-855-g19e2a2be-1focal [239 kB] 2024-09-21T15:12:41.751 INFO:teuthology.orchestra.run.smithi142.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-855-g19e2a2be-1focal [143 kB] 2024-09-21T15:12:41.752 INFO:teuthology.orchestra.run.smithi142.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-855-g19e2a2be-1focal [7,015 kB] 2024-09-21T15:12:41.756 INFO:teuthology.orchestra.run.smithi146.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-855-g19e2a2be-1focal [367 kB] 2024-09-21T15:12:41.763 INFO:teuthology.orchestra.run.smithi146.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-855-g19e2a2be-1focal [6,627 kB] 2024-09-21T15:12:41.804 INFO:teuthology.orchestra.run.smithi146.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-21T15:12:41.804 INFO:teuthology.orchestra.run.smithi146.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-21T15:12:41.885 INFO:teuthology.orchestra.run.smithi146.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-21T15:12:41.890 INFO:teuthology.orchestra.run.smithi146.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-21T15:12:41.894 INFO:teuthology.orchestra.run.smithi146.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-21T15:12:41.897 INFO:teuthology.orchestra.run.smithi146.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-21T15:12:41.898 INFO:teuthology.orchestra.run.smithi146.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-21T15:12:41.901 INFO:teuthology.orchestra.run.smithi146.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-21T15:12:41.903 INFO:teuthology.orchestra.run.smithi146.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-21T15:12:41.905 INFO:teuthology.orchestra.run.smithi142.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-855-g19e2a2be-1focal [7,893 kB] 2024-09-21T15:12:41.976 INFO:teuthology.orchestra.run.smithi146.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-21T15:12:41.982 INFO:teuthology.orchestra.run.smithi146.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-21T15:12:41.984 INFO:teuthology.orchestra.run.smithi146.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-21T15:12:42.087 INFO:teuthology.orchestra.run.smithi142.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-855-g19e2a2be-1focal [13.9 kB] 2024-09-21T15:12:42.087 INFO:teuthology.orchestra.run.smithi142.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-855-g19e2a2be-1focal [46.9 MB] 2024-09-21T15:12:42.146 INFO:teuthology.orchestra.run.smithi146.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-21T15:12:42.147 INFO:teuthology.orchestra.run.smithi146.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-21T15:12:42.147 INFO:teuthology.orchestra.run.smithi146.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-21T15:12:42.148 INFO:teuthology.orchestra.run.smithi146.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-21T15:12:42.160 INFO:teuthology.orchestra.run.smithi146.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T15:12:42.161 INFO:teuthology.orchestra.run.smithi146.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-21T15:12:42.162 INFO:teuthology.orchestra.run.smithi146.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-21T15:12:42.214 INFO:teuthology.orchestra.run.smithi146.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-21T15:12:42.214 INFO:teuthology.orchestra.run.smithi146.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-21T15:12:42.296 INFO:teuthology.orchestra.run.smithi067.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-855-g19e2a2be-1focal [113 kB] 2024-09-21T15:12:42.338 INFO:teuthology.orchestra.run.smithi067.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-855-g19e2a2be-1focal [478 kB] 2024-09-21T15:12:42.369 INFO:teuthology.orchestra.run.smithi067.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-855-g19e2a2be-1focal [25.2 MB] 2024-09-21T15:12:42.496 INFO:teuthology.orchestra.run.smithi146.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-855-g19e2a2be-1focal [113 kB] 2024-09-21T15:12:42.579 INFO:teuthology.orchestra.run.smithi146.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-855-g19e2a2be-1focal [478 kB] 2024-09-21T15:12:42.588 INFO:teuthology.orchestra.run.smithi146.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-855-g19e2a2be-1focal [25.2 MB] 2024-09-21T15:12:43.625 INFO:teuthology.orchestra.run.smithi142.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-855-g19e2a2be-1focal [120 kB] 2024-09-21T15:12:43.627 INFO:teuthology.orchestra.run.smithi142.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-855-g19e2a2be-1focal [37.4 kB] 2024-09-21T15:12:43.629 INFO:teuthology.orchestra.run.smithi142.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-855-g19e2a2be-1focal [13.5 MB] 2024-09-21T15:12:43.933 INFO:teuthology.orchestra.run.smithi142.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-855-g19e2a2be-1focal [108 kB] 2024-09-21T15:12:43.934 INFO:teuthology.orchestra.run.smithi142.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 18.2.4-855-g19e2a2be-1focal [189 kB] 2024-09-21T15:12:44.053 INFO:teuthology.orchestra.run.smithi067.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-855-g19e2a2be-1focal [9,605 kB] 2024-09-21T15:12:44.151 INFO:teuthology.orchestra.run.smithi146.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-855-g19e2a2be-1focal [9,605 kB] 2024-09-21T15:12:44.294 INFO:teuthology.orchestra.run.smithi067.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-855-g19e2a2be-1focal [238 kB] 2024-09-21T15:12:44.296 INFO:teuthology.orchestra.run.smithi067.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-855-g19e2a2be-1focal [148 kB] 2024-09-21T15:12:44.310 INFO:teuthology.orchestra.run.smithi067.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-855-g19e2a2be-1focal [1,574 kB] 2024-09-21T15:12:44.361 INFO:teuthology.orchestra.run.smithi067.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-855-g19e2a2be-1focal [6,640 kB] 2024-09-21T15:12:44.436 INFO:teuthology.orchestra.run.smithi146.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-855-g19e2a2be-1focal [238 kB] 2024-09-21T15:12:44.452 INFO:teuthology.orchestra.run.smithi146.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-855-g19e2a2be-1focal [148 kB] 2024-09-21T15:12:44.454 INFO:teuthology.orchestra.run.smithi146.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-855-g19e2a2be-1focal [1,574 kB] 2024-09-21T15:12:44.508 INFO:teuthology.orchestra.run.smithi146.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-855-g19e2a2be-1focal [6,640 kB] 2024-09-21T15:12:44.571 INFO:teuthology.orchestra.run.smithi067.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-855-g19e2a2be-1focal [24.5 MB] 2024-09-21T15:12:44.592 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 199 MB in 4s (46.8 MB/s) 2024-09-21T15:12:44.727 INFO:teuthology.orchestra.run.smithi146.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-855-g19e2a2be-1focal [24.5 MB] 2024-09-21T15:12:44.833 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-21T15:12:44.892 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-21T15:12:44.896 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-21T15:12:45.006 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-21T15:12:45.394 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T15:12:45.410 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-21T15:12:45.470 INFO:teuthology.orchestra.run.smithi067.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-855-g19e2a2be-1focal [13.7 kB] 2024-09-21T15:12:45.471 INFO:teuthology.orchestra.run.smithi067.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-855-g19e2a2be-1focal [856 kB] 2024-09-21T15:12:45.486 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-21T15:12:45.487 INFO:teuthology.orchestra.run.smithi067.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-855-g19e2a2be-1focal [2,295 kB] 2024-09-21T15:12:45.503 INFO:teuthology.orchestra.run.smithi067.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-855-g19e2a2be-1focal [239 kB] 2024-09-21T15:12:45.505 INFO:teuthology.orchestra.run.smithi067.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-855-g19e2a2be-1focal [143 kB] 2024-09-21T15:12:45.506 INFO:teuthology.orchestra.run.smithi067.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-855-g19e2a2be-1focal [7,015 kB] 2024-09-21T15:12:45.607 INFO:teuthology.orchestra.run.smithi146.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-855-g19e2a2be-1focal [13.7 kB] 2024-09-21T15:12:45.608 INFO:teuthology.orchestra.run.smithi146.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-855-g19e2a2be-1focal [856 kB] 2024-09-21T15:12:45.619 INFO:teuthology.orchestra.run.smithi146.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-855-g19e2a2be-1focal [2,295 kB] 2024-09-21T15:12:45.662 INFO:teuthology.orchestra.run.smithi146.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-855-g19e2a2be-1focal [239 kB] 2024-09-21T15:12:45.680 INFO:teuthology.orchestra.run.smithi146.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-855-g19e2a2be-1focal [143 kB] 2024-09-21T15:12:45.683 INFO:teuthology.orchestra.run.smithi146.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-855-g19e2a2be-1focal [7,015 kB] 2024-09-21T15:12:45.700 INFO:teuthology.orchestra.run.smithi067.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-855-g19e2a2be-1focal [7,893 kB] 2024-09-21T15:12:45.830 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T15:12:45.848 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-21T15:12:45.888 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-21T15:12:45.916 INFO:teuthology.orchestra.run.smithi146.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-855-g19e2a2be-1focal [7,893 kB] 2024-09-21T15:12:45.952 INFO:teuthology.orchestra.run.smithi067.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-855-g19e2a2be-1focal [13.9 kB] 2024-09-21T15:12:45.952 INFO:teuthology.orchestra.run.smithi067.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-855-g19e2a2be-1focal [46.9 MB] 2024-09-21T15:12:46.195 INFO:teuthology.orchestra.run.smithi146.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-855-g19e2a2be-1focal [13.9 kB] 2024-09-21T15:12:46.195 INFO:teuthology.orchestra.run.smithi146.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-855-g19e2a2be-1focal [46.9 MB] 2024-09-21T15:12:46.416 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T15:12:46.433 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:46.485 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:47.036 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T15:12:47.053 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:47.094 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:47.456 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T15:12:47.475 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:47.539 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:47.641 INFO:teuthology.orchestra.run.smithi067.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-855-g19e2a2be-1focal [120 kB] 2024-09-21T15:12:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-855-g19e2a2be-1focal [37.4 kB] 2024-09-21T15:12:47.645 INFO:teuthology.orchestra.run.smithi067.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-855-g19e2a2be-1focal [13.5 MB] 2024-09-21T15:12:47.896 INFO:teuthology.orchestra.run.smithi146.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-855-g19e2a2be-1focal [120 kB] 2024-09-21T15:12:47.898 INFO:teuthology.orchestra.run.smithi146.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-855-g19e2a2be-1focal [37.4 kB] 2024-09-21T15:12:47.899 INFO:teuthology.orchestra.run.smithi146.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-855-g19e2a2be-1focal [13.5 MB] 2024-09-21T15:12:47.902 INFO:teuthology.orchestra.run.smithi067.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-855-g19e2a2be-1focal [108 kB] 2024-09-21T15:12:47.904 INFO:teuthology.orchestra.run.smithi067.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 18.2.4-855-g19e2a2be-1focal [189 kB] 2024-09-21T15:12:47.967 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-21T15:12:47.985 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-21T15:12:48.046 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-21T15:12:48.165 INFO:teuthology.orchestra.run.smithi146.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-855-g19e2a2be-1focal [108 kB] 2024-09-21T15:12:48.166 INFO:teuthology.orchestra.run.smithi146.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 18.2.4-855-g19e2a2be-1focal [189 kB] 2024-09-21T15:12:48.437 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../007-librbd1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:48.562 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 199 MB in 8s (24.5 MB/s) 2024-09-21T15:12:48.587 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librbd1 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-21T15:12:48.761 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-21T15:12:48.817 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-21T15:12:48.822 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-21T15:12:48.888 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 199 MB in 8s (23.6 MB/s) 2024-09-21T15:12:48.901 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-21T15:12:49.058 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-21T15:12:49.114 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-21T15:12:49.119 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-21T15:12:49.207 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-21T15:12:49.221 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T15:12:49.240 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-21T15:12:49.292 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../008-librados2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:49.304 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-21T15:12:49.441 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librados2 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-21T15:12:49.518 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T15:12:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-21T15:12:49.565 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T15:12:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-21T15:12:49.584 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-21T15:12:49.615 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-21T15:12:49.930 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T15:12:49.947 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-21T15:12:49.988 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-21T15:12:50.102 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T15:12:50.121 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:50.171 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:50.237 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T15:12:50.253 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:50.295 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:50.441 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T15:12:50.459 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:50.509 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:50.631 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rados. 2024-09-21T15:12:50.648 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../010-python3-rados_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:50.655 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T15:12:50.672 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:50.705 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:50.706 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:50.983 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T15:12:50.985 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T15:12:51.001 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:51.003 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:51.035 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:51.043 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:51.076 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T15:12:51.095 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:12:51.134 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:51.430 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T15:12:51.446 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-21T15:12:51.455 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-21T15:12:51.461 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T15:12:51.472 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-21T15:12:51.479 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:51.488 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:12:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-21T15:12:51.519 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:51.847 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T15:12:51.867 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:12:51.875 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-21T15:12:51.882 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../007-librbd1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:51.892 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-21T15:12:51.905 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:51.933 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-21T15:12:51.983 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librbd1 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-21T15:12:52.292 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T15:12:52.304 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../007-librbd1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:52.308 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-21T15:12:52.350 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-21T15:12:52.430 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librbd1 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-21T15:12:52.653 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T15:12:52.669 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:52.698 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../008-librados2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:52.744 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:52.820 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librados2 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-21T15:12:53.097 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-21T15:12:53.114 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-21T15:12:53.118 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../008-librados2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:53.155 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-21T15:12:53.242 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librados2 (18.2.4-855-g19e2a2be-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-21T15:12:53.542 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-21T15:12:53.557 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-21T15:12:53.599 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T15:12:53.619 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:53.624 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-21T15:12:53.666 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:53.963 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T15:12:53.982 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:53.993 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rados. 2024-09-21T15:12:54.010 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../010-python3-rados_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:54.011 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T15:12:54.031 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-21T15:12:54.037 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:54.051 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:54.069 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-21T15:12:54.357 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rados. 2024-09-21T15:12:54.363 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T15:12:54.375 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../010-python3-rados_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:54.378 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:12:54.412 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:54.415 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librgw2. 2024-09-21T15:12:54.464 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../019-librgw2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:54.530 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:54.665 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T15:12:54.681 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:54.718 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T15:12:54.723 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:54.733 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:12:54.767 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:55.003 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T15:12:55.019 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:55.061 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:55.084 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T15:12:55.101 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:12:55.142 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:55.391 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T15:12:55.408 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:55.441 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:55.461 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T15:12:55.479 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:12:55.495 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T15:12:55.511 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-21T15:12:55.553 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-21T15:12:55.605 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:55.797 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T15:12:55.815 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:55.855 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:55.869 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liboath0. 2024-09-21T15:12:55.885 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-21T15:12:55.927 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-21T15:12:55.967 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T15:12:55.984 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-21T15:12:56.025 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-21T15:12:56.175 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-21T15:12:56.195 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-21T15:12:56.230 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T15:12:56.233 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-21T15:12:56.248 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:56.253 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T15:12:56.269 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:56.288 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:56.311 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:56.569 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-21T15:12:56.588 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-21T15:12:56.627 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-21T15:12:56.630 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-21T15:12:56.648 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-21T15:12:56.689 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-21T15:12:56.708 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-common. 2024-09-21T15:12:56.724 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../023-ceph-common_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:56.980 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T15:12:56.997 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-21T15:12:57.042 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-21T15:12:57.060 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-21T15:12:57.100 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-21T15:12:57.132 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-21T15:12:57.436 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T15:12:57.452 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-21T15:12:57.494 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-21T15:12:57.500 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librgw2. 2024-09-21T15:12:57.519 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../019-librgw2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:57.549 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:57.814 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librgw2. 2024-09-21T15:12:57.831 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../019-librgw2_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:57.863 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:58.394 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T15:12:58.412 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:58.452 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:58.749 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T15:12:58.766 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:58.807 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:58.813 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liboath0. 2024-09-21T15:12:58.832 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-21T15:12:58.871 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-21T15:12:59.151 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liboath0. 2024-09-21T15:12:59.168 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-21T15:12:59.207 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T15:12:59.209 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-21T15:12:59.223 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:59.247 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-base. 2024-09-21T15:12:59.265 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../024-ceph-base_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:59.265 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:59.307 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:59.496 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T15:12:59.511 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:59.545 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:12:59.635 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-common. 2024-09-21T15:12:59.651 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../023-ceph-common_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:12:59.685 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:00.007 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-common. 2024-09-21T15:13:00.024 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../023-ceph-common_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:00.065 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:00.635 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T15:13:00.655 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-21T15:13:00.692 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-21T15:13:01.104 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T15:13:01.121 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-21T15:13:01.162 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-21T15:13:01.528 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T15:13:01.547 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-21T15:13:01.599 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-21T15:13:01.927 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T15:13:01.943 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-21T15:13:01.985 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-21T15:13:02.148 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-base. 2024-09-21T15:13:02.165 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../024-ceph-base_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:02.217 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:02.355 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-mako. 2024-09-21T15:13:02.374 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-21T15:13:02.412 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-21T15:13:02.495 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-base. 2024-09-21T15:13:02.512 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../024-ceph-base_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:02.564 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:02.774 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T15:13:02.792 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T15:13:02.849 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T15:13:03.160 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T15:13:03.179 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-21T15:13:03.218 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-21T15:13:03.538 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webob. 2024-09-21T15:13:03.554 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-21T15:13:03.595 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T15:13:03.612 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-21T15:13:03.615 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-21T15:13:03.653 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-21T15:13:03.758 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T15:13:03.776 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-21T15:13:03.816 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-21T15:13:03.974 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-21T15:13:03.989 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T15:13:03.990 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-21T15:13:04.006 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-21T15:13:04.032 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-21T15:13:04.047 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-21T15:13:04.161 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T15:13:04.177 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-21T15:13:04.219 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-21T15:13:04.367 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T15:13:04.385 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-bs4. 2024-09-21T15:13:04.387 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-21T15:13:04.404 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-21T15:13:04.425 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-21T15:13:04.443 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-21T15:13:04.531 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T15:13:04.551 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-21T15:13:04.589 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-21T15:13:04.712 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T15:13:04.729 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-21T15:13:04.788 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-21T15:13:04.804 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T15:13:04.823 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-21T15:13:04.875 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T15:13:04.895 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-21T15:13:04.899 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-21T15:13:04.933 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-21T15:13:05.081 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-mako. 2024-09-21T15:13:05.097 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-21T15:13:05.139 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-21T15:13:05.203 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-mako. 2024-09-21T15:13:05.223 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-21T15:13:05.249 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T15:13:05.261 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-21T15:13:05.265 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-21T15:13:05.307 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-21T15:13:05.475 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T15:13:05.493 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T15:13:05.534 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T15:13:05.597 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T15:13:05.615 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T15:13:05.655 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T15:13:05.669 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-paste. 2024-09-21T15:13:05.687 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-21T15:13:05.727 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-21T15:13:05.820 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T15:13:05.839 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-21T15:13:05.878 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-21T15:13:05.975 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T15:13:05.992 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-21T15:13:06.033 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-21T15:13:06.085 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T15:13:06.101 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-21T15:13:06.169 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-21T15:13:06.206 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webob. 2024-09-21T15:13:06.222 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-21T15:13:06.255 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-21T15:13:06.336 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webob. 2024-09-21T15:13:06.352 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-21T15:13:06.394 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-21T15:13:06.464 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T15:13:06.483 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-21T15:13:06.522 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-21T15:13:06.575 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-21T15:13:06.591 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-21T15:13:06.666 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-21T15:13:06.722 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-21T15:13:06.742 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-21T15:13:06.780 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-21T15:13:06.850 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T15:13:06.866 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-21T15:13:06.933 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-21T15:13:06.969 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-bs4. 2024-09-21T15:13:06.985 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-21T15:13:07.028 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-21T15:13:07.083 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bs4. 2024-09-21T15:13:07.099 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-21T15:13:07.141 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-21T15:13:07.286 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T15:13:07.302 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-21T15:13:07.344 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-21T15:13:07.364 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T15:13:07.381 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-21T15:13:07.413 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-21T15:13:07.503 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T15:13:07.518 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-21T15:13:07.564 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-21T15:13:07.766 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T15:13:07.784 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-21T15:13:07.823 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-21T15:13:07.897 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T15:13:07.913 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-21T15:13:07.955 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-21T15:13:08.097 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T15:13:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-21T15:13:08.155 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-21T15:13:08.160 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T15:13:08.179 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:08.229 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:08.241 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-paste. 2024-09-21T15:13:08.257 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-21T15:13:08.299 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-21T15:13:08.458 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-paste. 2024-09-21T15:13:08.474 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-21T15:13:08.538 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-21T15:13:08.628 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T15:13:08.634 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T15:13:08.647 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-21T15:13:08.652 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:08.686 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-21T15:13:08.700 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:08.844 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T15:13:08.861 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-21T15:13:08.902 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-21T15:13:09.014 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T15:13:09.029 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-21T15:13:09.065 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T15:13:09.076 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-21T15:13:09.082 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-21T15:13:09.123 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-21T15:13:09.247 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T15:13:09.263 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-21T15:13:09.305 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-21T15:13:09.363 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T15:13:09.383 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-21T15:13:09.413 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-21T15:13:09.485 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cryptography. 2024-09-21T15:13:09.501 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-21T15:13:09.543 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-21T15:13:09.591 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T15:13:09.611 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-21T15:13:09.650 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-21T15:13:09.733 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T15:13:09.749 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-21T15:13:09.791 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-21T15:13:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T15:13:09.972 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-21T15:13:09.995 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T15:13:10.011 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-21T15:13:10.015 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-21T15:13:10.062 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-21T15:13:10.129 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T15:13:10.201 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-21T15:13:10.201 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-21T15:13:10.341 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T15:13:10.358 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-21T15:13:10.432 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T15:13:10.432 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-21T15:13:10.449 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:10.515 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:10.516 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T15:13:10.531 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:10.565 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:10.706 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T15:13:10.722 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:10.764 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:10.920 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T15:13:10.936 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:11.053 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:11.068 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T15:13:11.084 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:11.110 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T15:13:11.126 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:11.126 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:11.158 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:11.486 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T15:13:11.502 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-21T15:13:11.544 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-21T15:13:11.855 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cryptography. 2024-09-21T15:13:11.875 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-21T15:13:11.914 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-21T15:13:12.039 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T15:13:12.059 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-21T15:13:12.097 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-21T15:13:12.113 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T15:13:12.130 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:12.171 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:12.284 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T15:13:12.303 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-21T15:13:12.342 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-21T15:13:12.377 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cryptography. 2024-09-21T15:13:12.395 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-21T15:13:12.437 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-21T15:13:12.696 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T15:13:12.713 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:12.753 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:12.807 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T15:13:12.824 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-21T15:13:12.866 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-21T15:13:13.201 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T15:13:13.217 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:13.267 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:13.787 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T15:13:13.804 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:13.836 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:14.024 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T15:13:14.041 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:14.082 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:14.583 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph. 2024-09-21T15:13:14.601 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../051-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:14.641 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:14.765 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T15:13:14.781 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:14.823 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:15.002 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T15:13:15.021 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:15.035 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T15:13:15.053 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:15.060 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:15.093 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:15.555 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T15:13:15.572 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:15.613 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:16.200 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package cephadm. 2024-09-21T15:13:16.219 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../054-cephadm_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:16.282 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:16.627 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T15:13:16.643 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-21T15:13:16.685 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-21T15:13:17.001 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph. 2024-09-21T15:13:17.019 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../051-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:17.059 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:17.097 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T15:13:17.113 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-21T15:13:17.155 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-21T15:13:17.364 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph. 2024-09-21T15:13:17.380 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../051-ceph_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:17.388 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T15:13:17.403 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:17.413 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:17.437 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:17.524 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T15:13:17.543 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:17.582 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:17.741 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T15:13:17.757 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:17.799 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:17.866 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T15:13:17.884 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:17.923 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:17.936 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jwt. 2024-09-21T15:13:17.953 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-21T15:13:17.993 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-21T15:13:18.219 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T15:13:18.239 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:18.268 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:18.355 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-attr. 2024-09-21T15:13:18.372 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-21T15:13:18.413 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-21T15:13:18.435 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package cephadm. 2024-09-21T15:13:18.452 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../054-cephadm_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:18.493 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:18.804 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T15:13:18.823 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-21T15:13:18.833 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T15:13:18.847 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephadm. 2024-09-21T15:13:18.850 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-21T15:13:18.862 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-21T15:13:18.867 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../054-cephadm_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:18.891 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-21T15:13:18.904 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:19.232 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T15:13:19.235 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-py. 2024-09-21T15:13:19.252 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-21T15:13:19.253 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-21T15:13:19.265 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T15:13:19.284 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-21T15:13:19.290 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-21T15:13:19.293 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-21T15:13:19.323 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-21T15:13:19.642 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T15:13:19.651 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T15:13:19.660 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:19.669 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-21T15:13:19.673 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-21T15:13:19.691 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-21T15:13:19.700 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:19.709 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-21T15:13:19.723 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-21T15:13:20.051 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-packaging. 2024-09-21T15:13:20.054 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jwt. 2024-09-21T15:13:20.062 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T15:13:20.069 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-21T15:13:20.069 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-21T15:13:20.078 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:20.112 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-21T15:13:20.120 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:20.134 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-21T15:13:20.432 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-attr. 2024-09-21T15:13:20.445 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T15:13:20.448 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-21T15:13:20.464 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-21T15:13:20.482 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jwt. 2024-09-21T15:13:20.490 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-21T15:13:20.500 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-21T15:13:20.503 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-21T15:13:20.540 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-21T15:13:20.834 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T15:13:20.852 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-21T15:13:20.852 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-attr. 2024-09-21T15:13:20.856 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T15:13:20.869 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-21T15:13:20.875 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-21T15:13:20.892 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-21T15:13:20.911 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-21T15:13:20.914 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-21T15:13:21.195 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-py. 2024-09-21T15:13:21.212 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-21T15:13:21.230 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T15:13:21.251 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-21T15:13:21.253 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-21T15:13:21.276 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T15:13:21.288 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-21T15:13:21.295 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T15:13:21.355 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T15:13:21.573 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-21T15:13:21.589 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-21T15:13:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-py. 2024-09-21T15:13:21.617 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-21T15:13:21.639 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-21T15:13:21.653 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-underscore. 2024-09-21T15:13:21.658 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-21T15:13:21.669 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-21T15:13:21.711 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-21T15:13:21.942 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-packaging. 2024-09-21T15:13:21.958 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-21T15:13:22.003 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-21T15:13:22.014 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-21T15:13:22.020 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-21T15:13:22.029 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-21T15:13:22.050 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-21T15:13:22.061 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-21T15:13:22.072 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-21T15:13:22.371 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T15:13:22.388 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-21T15:13:22.397 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-packaging. 2024-09-21T15:13:22.413 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-21T15:13:22.429 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-21T15:13:22.455 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-21T15:13:22.483 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-routes. 2024-09-21T15:13:22.498 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-21T15:13:22.541 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-21T15:13:22.724 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T15:13:22.743 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-21T15:13:22.757 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T15:13:22.774 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-21T15:13:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-21T15:13:22.816 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-21T15:13:22.894 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T15:13:22.910 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:22.968 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:23.102 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T15:13:23.118 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T15:13:23.118 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T15:13:23.137 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-21T15:13:23.160 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T15:13:23.176 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-21T15:13:23.412 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjs-underscore. 2024-09-21T15:13:23.429 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-21T15:13:23.470 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-21T15:13:23.513 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T15:13:23.530 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T15:13:23.588 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T15:13:23.781 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-21T15:13:23.798 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-21T15:13:23.824 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjs-underscore. 2024-09-21T15:13:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-21T15:13:23.843 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-21T15:13:23.882 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-21T15:13:24.159 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-routes. 2024-09-21T15:13:24.178 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-21T15:13:24.201 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-21T15:13:24.216 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-21T15:13:24.217 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-21T15:13:24.259 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-21T15:13:24.399 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-decorator. 2024-09-21T15:13:24.415 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-21T15:13:24.449 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-21T15:13:24.545 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T15:13:24.563 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:24.603 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:24.603 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-routes. 2024-09-21T15:13:24.619 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-21T15:13:24.661 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-21T15:13:24.768 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-21T15:13:24.787 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-21T15:13:24.851 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-21T15:13:25.023 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T15:13:25.039 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:25.081 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:25.188 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-scipy. 2024-09-21T15:13:25.205 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-21T15:13:25.245 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-21T15:13:25.975 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-decorator. 2024-09-21T15:13:25.991 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-21T15:13:26.032 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-21T15:13:26.335 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-21T15:13:26.352 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-21T15:13:26.393 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-21T15:13:26.461 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-decorator. 2024-09-21T15:13:26.478 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-21T15:13:26.519 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-21T15:13:26.638 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-21T15:13:26.654 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-21T15:13:26.696 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:26.721 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-scipy. 2024-09-21T15:13:26.738 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-21T15:13:26.779 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-21T15:13:26.864 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-21T15:13:26.881 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-21T15:13:26.922 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-21T15:13:27.266 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-scipy. 2024-09-21T15:13:27.275 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T15:13:27.283 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-21T15:13:27.295 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-21T15:13:27.324 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-21T15:13:27.358 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-21T15:13:27.745 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T15:13:27.762 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-21T15:13:27.803 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:28.329 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-21T15:13:28.346 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-21T15:13:28.387 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:28.554 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T15:13:28.570 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:28.629 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:28.709 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-21T15:13:28.728 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-21T15:13:28.758 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:28.966 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T15:13:28.982 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-21T15:13:29.024 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-21T15:13:29.254 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T15:13:29.270 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-21T15:13:29.312 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-21T15:13:29.395 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T15:13:29.411 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-21T15:13:29.453 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:29.718 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T15:13:29.735 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-21T15:13:29.776 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:29.857 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T15:13:29.876 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-21T15:13:29.915 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-21T15:13:30.186 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T15:13:30.205 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:30.238 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T15:13:30.244 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:30.254 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-21T15:13:30.296 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-21T15:13:30.571 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T15:13:30.592 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:30.629 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:30.691 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T15:13:30.707 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-21T15:13:30.749 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-21T15:13:31.221 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T15:13:31.240 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-21T15:13:31.279 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-21T15:13:31.364 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T15:13:31.381 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-21T15:13:31.422 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-21T15:13:31.666 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T15:13:31.683 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-21T15:13:31.724 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-21T15:13:31.727 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T15:13:31.745 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-21T15:13:31.765 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T15:13:31.782 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-21T15:13:31.785 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-21T15:13:31.823 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-21T15:13:32.122 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T15:13:32.126 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T15:13:32.129 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-blinker. 2024-09-21T15:13:32.138 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-21T15:13:32.144 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-21T15:13:32.146 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-21T15:13:32.180 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-21T15:13:32.187 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-21T15:13:32.192 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-21T15:13:32.479 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T15:13:32.499 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-21T15:13:32.524 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-21T15:13:32.537 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-21T15:13:32.541 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-21T15:13:32.581 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-21T15:13:32.622 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T15:13:32.638 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-21T15:13:32.680 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-21T15:13:32.942 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T15:13:32.958 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-21T15:13:32.960 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T15:13:32.972 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T15:13:32.977 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-21T15:13:32.990 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-21T15:13:32.992 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-21T15:13:33.018 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-21T15:13:33.030 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-21T15:13:33.287 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-blinker. 2024-09-21T15:13:33.307 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-21T15:13:33.330 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T15:13:33.345 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-21T15:13:33.348 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-21T15:13:33.383 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-21T15:13:33.409 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T15:13:33.428 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-21T15:13:33.467 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-21T15:13:33.632 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-21T15:13:33.649 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-21T15:13:33.690 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-21T15:13:33.712 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-blinker. 2024-09-21T15:13:33.729 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-21T15:13:33.770 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-21T15:13:33.786 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T15:13:33.805 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-21T15:13:33.844 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-21T15:13:34.044 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T15:13:34.061 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-21T15:13:34.098 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-21T15:13:34.102 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-21T15:13:34.117 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-21T15:13:34.158 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-21T15:13:34.417 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T15:13:34.418 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T15:13:34.433 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-21T15:13:34.437 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:34.474 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-21T15:13:34.476 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:34.503 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T15:13:34.521 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-21T15:13:34.561 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-21T15:13:34.820 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T15:13:34.820 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T15:13:34.837 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-21T15:13:34.840 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-21T15:13:34.895 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-21T15:13:34.897 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T15:13:34.912 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-21T15:13:34.914 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-21T15:13:34.947 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-21T15:13:35.251 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T15:13:35.272 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-21T15:13:35.274 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T15:13:35.292 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-21T15:13:35.309 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-21T15:13:35.332 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:13:35.408 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T15:13:35.425 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:35.478 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:35.643 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package jq. 2024-09-21T15:13:35.660 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-21T15:13:35.718 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:13:35.806 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T15:13:35.821 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-21T15:13:35.847 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T15:13:35.862 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-21T15:13:35.863 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:35.905 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:35.979 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package socat. 2024-09-21T15:13:35.995 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-21T15:13:36.037 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-21T15:13:36.215 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T15:13:36.233 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-21T15:13:36.266 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T15:13:36.273 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:13:36.285 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-21T15:13:36.324 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-21T15:13:36.373 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T15:13:36.391 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-21T15:13:36.456 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-21T15:13:36.543 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package jq. 2024-09-21T15:13:36.562 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-21T15:13:36.600 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:13:36.685 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T15:13:36.704 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-21T15:13:36.743 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:13:36.836 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-test. 2024-09-21T15:13:36.854 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package socat. 2024-09-21T15:13:36.856 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../094-ceph-test_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:36.870 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-21T15:13:36.894 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:36.911 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-21T15:13:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package jq. 2024-09-21T15:13:37.032 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-21T15:13:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:13:37.257 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T15:13:37.273 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-21T15:13:37.314 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-21T15:13:37.349 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package socat. 2024-09-21T15:13:37.368 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-21T15:13:37.407 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-21T15:13:37.652 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-test. 2024-09-21T15:13:37.672 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../094-ceph-test_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:37.710 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:37.735 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T15:13:37.753 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-21T15:13:37.810 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-21T15:13:38.150 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-test. 2024-09-21T15:13:38.168 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../094-ceph-test_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:38.209 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:41.073 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T15:13:41.092 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:41.123 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:41.446 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-21T15:13:41.465 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-21T15:13:41.504 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:41.839 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T15:13:41.857 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:41.889 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:42.066 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T15:13:42.083 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:42.124 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:42.151 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-21T15:13:42.170 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-21T15:13:42.210 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:42.330 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T15:13:42.346 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-855-g19e2a2be-1focal_all.deb ... 2024-09-21T15:13:42.387 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:42.561 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T15:13:42.580 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-21T15:13:42.639 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-21T15:13:42.669 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-21T15:13:42.689 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-21T15:13:42.715 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T15:13:42.727 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:42.735 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:42.773 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:42.993 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T15:13:43.010 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-21T15:13:43.076 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-21T15:13:43.202 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T15:13:43.219 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-21T15:13:43.247 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T15:13:43.265 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:43.278 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-21T15:13:43.305 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:43.454 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-21T15:13:43.471 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-21T15:13:43.512 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:43.573 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T15:13:43.590 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-21T15:13:43.634 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-21T15:13:43.659 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T15:13:43.677 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-21T15:13:43.743 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-21T15:13:44.029 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-21T15:13:44.046 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-21T15:13:44.073 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T15:13:44.087 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:44.092 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-21T15:13:44.132 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-21T15:13:44.501 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-21T15:13:44.519 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-21T15:13:44.559 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:45.226 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-21T15:13:45.244 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-21T15:13:45.301 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-21T15:13:45.662 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webencodings. 2024-09-21T15:13:45.681 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-21T15:13:45.714 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-21T15:13:45.720 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-21T15:13:45.732 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-21T15:13:45.805 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-21T15:13:46.073 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-html5lib. 2024-09-21T15:13:46.093 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-21T15:13:46.131 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-21T15:13:46.141 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webencodings. 2024-09-21T15:13:46.159 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-21T15:13:46.176 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-21T15:13:46.195 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-21T15:13:46.199 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-21T15:13:46.233 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-21T15:13:46.502 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-21T15:13:46.502 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-html5lib. 2024-09-21T15:13:46.518 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-21T15:13:46.522 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-21T15:13:46.560 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-21T15:13:46.578 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-21T15:13:46.581 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webencodings. 2024-09-21T15:13:46.601 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-21T15:13:46.631 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-21T15:13:46.897 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-21T15:13:46.914 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-21T15:13:46.934 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-html5lib. 2024-09-21T15:13:46.951 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-21T15:13:46.955 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-21T15:13:46.991 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-21T15:13:47.098 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T15:13:47.115 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T15:13:47.148 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T15:13:47.370 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-21T15:13:47.387 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-21T15:13:47.428 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-21T15:13:47.441 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T15:13:47.458 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T15:13:47.499 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T15:13:47.510 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T15:13:47.527 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-21T15:13:47.601 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-21T15:13:47.789 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T15:13:47.809 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-21T15:13:47.847 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-21T15:13:47.864 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T15:13:47.884 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T15:13:47.922 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T15:13:47.954 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T15:13:47.973 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-21T15:13:48.012 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-21T15:13:48.175 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T15:13:48.192 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-21T15:13:48.225 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-21T15:13:48.226 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T15:13:48.244 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-21T15:13:48.283 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-21T15:13:48.390 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T15:13:48.407 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-21T15:13:48.448 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-21T15:13:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T15:13:48.540 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-21T15:13:48.577 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-21T15:13:48.639 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T15:13:48.657 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-21T15:13:48.697 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-21T15:13:48.759 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T15:13:48.778 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-21T15:13:48.817 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-21T15:13:48.896 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T15:13:48.917 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-21T15:13:48.954 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-21T15:13:49.025 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T15:13:49.042 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-21T15:13:49.083 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-21T15:13:49.361 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T15:13:49.378 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-21T15:13:49.418 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-21T15:13:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package radosgw. 2024-09-21T15:13:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../110-radosgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:49.649 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package radosgw. 2024-09-21T15:13:49.669 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../110-radosgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:49.706 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:50.081 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package radosgw. 2024-09-21T15:13:50.099 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../110-radosgw_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:50.139 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:51.055 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T15:13:51.072 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:51.113 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:51.185 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T15:13:51.204 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:51.243 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:51.366 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-nbd. 2024-09-21T15:13:51.385 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../112-rbd-nbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:51.423 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:51.493 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T15:13:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:51.521 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package rbd-nbd. 2024-09-21T15:13:51.538 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../112-rbd-nbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:51.570 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:51.576 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:51.714 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-21T15:13:51.837 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-nbd. 2024-09-21T15:13:51.857 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../112-rbd-nbd_18.2.4-855-g19e2a2be-1focal_amd64.deb ... 2024-09-21T15:13:51.895 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:51.918 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-21T15:13:52.029 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-21T15:13:52.155 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-21T15:13:52.227 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-21T15:13:52.236 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-21T15:13:52.280 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-21T15:13:52.352 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-21T15:13:52.483 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:52.508 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-21T15:13:52.512 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-21T15:13:52.634 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-21T15:13:52.739 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:52.760 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-21T15:13:52.995 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:53.034 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user cephadm....done 2024-09-21T15:13:53.204 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:53.290 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user cephadm....done 2024-09-21T15:13:53.332 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-21T15:13:53.486 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:53.604 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user cephadm....done 2024-09-21T15:13:53.612 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-21T15:13:53.683 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T15:13:53.767 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:53.893 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-21T15:13:53.947 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T15:13:53.959 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:54.085 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-21T15:13:54.211 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-21T15:13:54.237 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T15:13:54.281 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:54.407 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-21T15:13:54.513 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-21T15:13:54.555 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-21T15:13:54.575 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-21T15:13:54.672 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-21T15:13:54.798 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-21T15:13:54.910 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-21T15:13:54.948 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-21T15:13:55.125 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-21T15:13:55.249 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-21T15:13:55.354 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-21T15:13:55.527 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-21T15:13:55.534 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-21T15:13:55.638 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-21T15:13:55.827 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-21T15:13:55.828 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-21T15:13:55.923 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-21T15:13:56.119 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:56.146 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-21T15:13:56.241 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-21T15:13:56.439 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-21T15:13:56.446 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:56.567 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:56.571 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-21T15:13:56.765 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-argparse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:13:56.864 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-21T15:13:56.868 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:56.994 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-21T15:13:57.057 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-21T15:13:57.157 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-21T15:13:57.200 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-21T15:13:57.283 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-21T15:13:57.337 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-21T15:13:57.493 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-21T15:13:57.621 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-21T15:13:57.659 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-21T15:13:57.747 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-21T15:13:57.777 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-21T15:13:57.905 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-21T15:13:57.935 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-21T15:13:58.156 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-21T15:13:58.219 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-21T15:13:58.281 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-21T15:13:58.457 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-21T15:13:58.596 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-21T15:13:58.607 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-21T15:13:58.767 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-21T15:13:58.875 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-21T15:13:58.880 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-21T15:13:59.168 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-21T15:13:59.234 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-21T15:13:59.239 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-21T15:13:59.461 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-21T15:13:59.536 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-21T15:13:59.548 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-21T15:13:59.787 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-21T15:13:59.858 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-21T15:13:59.895 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-21T15:13:59.975 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-21T15:14:00.093 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-21T15:14:00.221 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-21T15:14:00.386 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-21T15:14:00.488 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-21T15:14:00.514 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-21T15:14:00.640 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-21T15:14:00.756 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-21T15:14:00.766 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-21T15:14:00.796 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-21T15:14:00.857 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-21T15:14:01.008 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-21T15:14:01.100 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-21T15:14:01.164 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-21T15:14:01.318 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-21T15:14:01.527 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-21T15:14:01.549 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-21T15:14:01.675 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:01.711 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-21T15:14:01.886 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-21T15:14:02.001 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:02.096 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-21T15:14:02.193 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-21T15:14:02.272 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-21T15:14:02.398 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:02.447 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-21T15:14:02.555 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-21T15:14:02.573 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:02.681 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-21T15:14:02.724 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:02.807 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-21T15:14:02.858 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-21T15:14:02.883 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:03.051 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-21T15:14:03.210 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-21T15:14:03.361 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-21T15:14:03.402 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-21T15:14:03.459 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-21T15:14:03.486 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-21T15:14:03.562 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-21T15:14:03.688 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-21T15:14:03.771 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-21T15:14:03.851 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:03.956 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-21T15:14:03.977 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-21T15:14:04.103 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-21T15:14:04.106 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:04.232 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-21T15:14:04.265 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:04.358 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-21T15:14:04.416 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-21T15:14:04.429 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-21T15:14:04.576 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-21T15:14:04.684 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-21T15:14:04.739 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-21T15:14:04.894 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-21T15:14:04.968 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-21T15:14:05.048 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-21T15:14:05.170 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-21T15:14:05.174 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T15:14:05.261 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-21T15:14:05.387 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T15:14:05.450 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-21T15:14:05.463 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-21T15:14:05.605 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T15:14:05.663 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-21T15:14:05.734 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-21T15:14:05.890 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-21T15:14:05.981 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-21T15:14:06.027 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-21T15:14:06.166 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-21T15:14:06.282 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-21T15:14:06.460 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-21T15:14:06.467 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-21T15:14:06.640 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-21T15:14:06.721 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-21T15:14:06.856 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-21T15:14:06.858 INFO:teuthology.orchestra.run.smithi142.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-21T15:14:06.901 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-21T15:14:07.052 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-21T15:14:07.128 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-21T15:14:07.182 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-21T15:14:07.254 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-21T15:14:07.378 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-21T15:14:07.550 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-21T15:14:07.614 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-21T15:14:07.676 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-21T15:14:07.755 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-21T15:14:07.785 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-21T15:14:07.881 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-21T15:14:07.949 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-21T15:14:08.007 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-21T15:14:08.058 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-21T15:14:08.111 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-21T15:14:08.226 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-21T15:14:08.341 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-21T15:14:08.420 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-21T15:14:08.553 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-21T15:14:08.659 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-21T15:14:08.853 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-21T15:14:09.901 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:14:10.018 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-21T15:14:10.198 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:14:10.324 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-21T15:14:10.350 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:14:10.395 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-21T15:14:10.501 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-21T15:14:10.687 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-21T15:14:10.700 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-21T15:14:10.878 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-21T15:14:10.989 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:11.010 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-21T15:14:11.114 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-21T15:14:11.212 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-21T15:14:11.294 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:11.420 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-21T15:14:11.449 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-21T15:14:11.505 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-21T15:14:11.606 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-21T15:14:11.746 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-21T15:14:11.784 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-21T15:14:11.974 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-21T15:14:12.090 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-21T15:14:12.143 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-21T15:14:12.301 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-21T15:14:12.486 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-21T15:14:12.499 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-21T15:14:12.702 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-21T15:14:12.834 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-21T15:14:13.020 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-21T15:14:13.508 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-21T15:14:13.634 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-21T15:14:13.751 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:14:13.902 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-21T15:14:13.914 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-21T15:14:14.040 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-21T15:14:14.076 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-21T15:14:14.166 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:14:14.204 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-21T15:14:14.235 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-21T15:14:14.292 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-21T15:14:14.338 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-21T15:14:14.361 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-21T15:14:14.512 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-21T15:14:14.618 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-21T15:14:14.639 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-21T15:14:14.744 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-21T15:14:14.813 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-21T15:14:14.914 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-21T15:14:14.932 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T15:14:15.064 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-21T15:14:15.240 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-21T15:14:15.258 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-21T15:14:15.525 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T15:14:15.652 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librados2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:15.713 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T15:14:15.778 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-21T15:14:15.851 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-21T15:14:15.904 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.030 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.040 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-21T15:14:16.231 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librados2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.241 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.363 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-21T15:14:16.367 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-21T15:14:16.408 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librados2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.497 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.534 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-21T15:14:16.598 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.660 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.660 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librgw2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.811 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-21T15:14:16.820 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libsqlite3-mod-ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.826 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:16.952 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-21T15:14:17.023 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs2 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.121 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librbd1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.150 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-21T15:14:17.255 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.287 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.356 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.413 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-21T15:14:17.458 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libradosstriper1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.568 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T15:14:17.568 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T15:14:17.584 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-21T15:14:17.731 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librbd1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.866 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.886 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librbd1 (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:17.983 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.004 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-modules-core (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.024 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.150 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.163 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.220 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T15:14:18.220 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T15:14:18.292 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-21T15:14:18.402 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T15:14:18.402 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T15:14:18.635 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.794 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.865 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs-dev (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:18.920 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-21T15:14:19.007 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rados (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.133 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-21T15:14:19.213 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.314 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.415 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.516 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.642 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.801 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.857 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:19.966 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.087 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.117 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.213 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-fuse (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.243 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.337 INFO:teuthology.orchestra.run.smithi146.stdout:Adding group ceph....done 2024-09-21T15:14:20.339 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-nbd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.419 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.466 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.570 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.625 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cephfs (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.680 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user ceph....done 2024-09-21T15:14:20.768 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-common (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:20.849 INFO:teuthology.orchestra.run.smithi146.stdout:Setting system user ceph properties....done 2024-09-21T15:14:20.870 INFO:teuthology.orchestra.run.smithi146.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T15:14:21.022 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T15:14:21.165 INFO:teuthology.orchestra.run.smithi142.stdout:Adding group ceph....done 2024-09-21T15:14:21.341 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T15:14:21.345 INFO:teuthology.orchestra.run.smithi067.stdout:Adding group ceph....done 2024-09-21T15:14:21.600 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user ceph....done 2024-09-21T15:14:21.755 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user ceph....done 2024-09-21T15:14:21.767 INFO:teuthology.orchestra.run.smithi142.stdout:Setting system user ceph properties....done 2024-09-21T15:14:21.791 INFO:teuthology.orchestra.run.smithi142.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T15:14:21.802 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:21.923 INFO:teuthology.orchestra.run.smithi067.stdout:Setting system user ceph properties....done 2024-09-21T15:14:21.928 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:21.940 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T15:14:21.948 INFO:teuthology.orchestra.run.smithi067.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T15:14:22.110 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T15:14:22.264 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T15:14:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:14:22.404 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:14:22.420 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T15:14:22.730 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:22.881 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:22.902 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-test (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:23.045 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up radosgw (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:23.180 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T15:14:23.367 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:14:23.368 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:14:23.510 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:14:23.510 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:14:23.615 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:23.816 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T15:14:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T15:14:23.955 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-base (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:24.235 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T15:14:24.251 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:24.347 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T15:14:24.473 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T15:14:24.474 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T15:14:24.685 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:24.840 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mds (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:24.917 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T15:14:24.917 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T15:14:24.918 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:25.079 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T15:14:25.080 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T15:14:25.353 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:25.386 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T15:14:25.386 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T15:14:25.525 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:25.595 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T15:14:25.595 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T15:14:25.762 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T15:14:25.763 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T15:14:25.820 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:25.946 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:26.081 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:26.088 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:26.227 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-osd (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:26.291 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T15:14:26.291 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T15:14:26.616 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T15:14:26.616 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T15:14:26.723 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:26.728 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T15:14:26.729 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T15:14:26.866 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:26.975 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.073 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.126 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.162 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-k8sevents (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.241 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.313 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.384 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.448 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mon (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:27.488 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-21T15:14:27.639 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-21T15:14:27.641 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T15:14:27.641 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T15:14:27.678 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T15:14:27.678 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T15:14:27.924 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-21T15:14:28.102 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.132 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-cephadm (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.278 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.283 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.404 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.409 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-dashboard (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.563 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.577 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-volume (18.2.4-855-g19e2a2be-1focal) ... 2024-09-21T15:14:28.916 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-21T15:14:28.946 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-21T15:14:29.093 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-21T15:14:29.123 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-21T15:14:29.386 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-21T15:14:29.433 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-21T15:14:30.785 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-21T15:14:31.356 INFO:teuthology.orchestra.run.smithi146.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T15:14:31.361 DEBUG:teuthology.parallel:result is None 2024-09-21T15:14:32.232 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-21T15:14:32.336 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-21T15:14:32.895 INFO:teuthology.orchestra.run.smithi142.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T15:14:32.899 DEBUG:teuthology.parallel:result is None 2024-09-21T15:14:32.942 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T15:14:32.946 DEBUG:teuthology.parallel:result is None 2024-09-21T15:14:32.947 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:14:33.095 DEBUG:teuthology.orchestra.run.smithi067:> dpkg-query -W -f '${Version}' ceph 2024-09-21T15:14:33.119 INFO:teuthology.orchestra.run.smithi067.stdout:18.2.4-855-g19e2a2be-1focal 2024-09-21T15:14:33.119 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855-g19e2a2be-1focal 2024-09-21T15:14:33.120 INFO:teuthology.task.install:The correct ceph version 18.2.4-855-g19e2a2be-1focal is installed. 2024-09-21T15:14:33.121 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:14:33.262 DEBUG:teuthology.orchestra.run.smithi142:> dpkg-query -W -f '${Version}' ceph 2024-09-21T15:14:33.287 INFO:teuthology.orchestra.run.smithi142.stdout:18.2.4-855-g19e2a2be-1focal 2024-09-21T15:14:33.287 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855-g19e2a2be-1focal 2024-09-21T15:14:33.287 INFO:teuthology.task.install:The correct ceph version 18.2.4-855-g19e2a2be-1focal is installed. 2024-09-21T15:14:33.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:14:33.429 DEBUG:teuthology.orchestra.run.smithi146:> dpkg-query -W -f '${Version}' ceph 2024-09-21T15:14:33.452 INFO:teuthology.orchestra.run.smithi146.stdout:18.2.4-855-g19e2a2be-1focal 2024-09-21T15:14:33.452 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855-g19e2a2be-1focal 2024-09-21T15:14:33.452 INFO:teuthology.task.install:The correct ceph version 18.2.4-855-g19e2a2be-1focal is installed. 2024-09-21T15:14:33.454 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T15:14:33.454 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:33.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T15:14:33.471 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:33.471 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T15:14:33.486 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:14:33.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T15:14:33.508 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T15:14:33.508 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:33.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T15:14:33.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T15:14:33.535 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:33.535 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T15:14:33.549 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T15:14:33.562 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:14:33.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T15:14:33.577 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T15:14:33.587 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T15:14:33.588 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:33.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T15:14:33.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T15:14:33.619 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:33.619 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T15:14:33.635 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T15:14:33.649 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:14:33.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T15:14:33.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T15:14:33.674 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T15:14:33.674 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:33.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T15:14:33.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T15:14:33.749 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:33.749 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T15:14:33.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T15:14:33.821 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:14:33.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T15:14:33.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T15:14:33.891 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T15:14:33.987 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T15:14:33.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-09-21T15:14:33.990 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /var/log/ceph 2024-09-21T15:14:33.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-09-21T15:14:34.008 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T15:14:34.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T15:14:34.011 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T15:14:34.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T15:14:34.067 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T15:14:34.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T15:14:34.069 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T15:14:34.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T15:14:34.138 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T15:14:34.138 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T15:14:34.138 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971', 'branch': 'reef', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/rbd_nbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909971', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 4623, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'sleep_before_teardown': 0, 'subset': '80/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi142.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_22:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T15:14:34.138 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T15:14:34.141 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T15:14:34.142 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T15:14:34.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T15:14:34.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T15:14:34.195 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T15:14:34.211 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:34.212 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T15:14:34.216 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T15:14:34.216 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 360 Links: 1 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-21 15:11:04.553621790 +0000 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-21 15:11:04.377625848 +0000 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-21 15:11:04.377625848 +0000 2024-09-21T15:14:34.223 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-21T15:14:34.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T15:14:34.236 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-21T15:14:34.236 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-21T15:14:34.236 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000364711 s, 1.4 MB/s 2024-09-21T15:14:34.238 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T15:14:34.290 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 368 Links: 1 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-21 15:11:04.973612104 +0000 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-21 15:11:04.765616901 +0000 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-21 15:11:04.765616901 +0000 2024-09-21T15:14:34.297 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-21T15:14:34.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T15:14:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-21T15:14:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-21T15:14:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000402712 s, 1.3 MB/s 2024-09-21T15:14:34.310 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T15:14:34.362 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-21 15:11:05.397602326 +0000 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-21 15:11:05.181607308 +0000 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-21 15:11:05.181607308 +0000 2024-09-21T15:14:34.370 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-21T15:14:34.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T15:14:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-21T15:14:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-21T15:14:34.384 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000357292 s, 1.4 MB/s 2024-09-21T15:14:34.385 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T15:14:34.393 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-09-21T15:14:34.444 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T15:14:34.444 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.445 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 378 Links: 1 2024-09-21T15:14:34.445 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.445 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-21 15:11:05.829592365 +0000 2024-09-21T15:14:34.445 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-21 15:11:05.613597345 +0000 2024-09-21T15:14:34.445 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-21 15:11:05.613597345 +0000 2024-09-21T15:14:34.445 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-09-21T15:14:34.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T15:14:34.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-21T15:14:34.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-21T15:14:34.458 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000408388 s, 1.3 MB/s 2024-09-21T15:14:34.460 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T15:14:34.468 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:34.468 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:34.468 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T15:14:34.475 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T15:14:34.475 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2024-09-21T15:14:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T15:14:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.482 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 369 Links: 1 2024-09-21T15:14:34.482 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.482 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-21 15:11:02.131316348 +0000 2024-09-21T15:14:34.482 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-21 15:11:01.923321146 +0000 2024-09-21T15:14:34.482 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-21 15:11:01.923321146 +0000 2024-09-21T15:14:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-21T15:14:34.483 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T15:14:34.495 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-21T15:14:34.495 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-21T15:14:34.496 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.00037277 s, 1.4 MB/s 2024-09-21T15:14:34.496 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T15:14:34.503 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2024-09-21T15:14:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T15:14:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.509 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-21T15:14:34.509 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.509 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-21 15:11:02.539306937 +0000 2024-09-21T15:14:34.509 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-21 15:11:02.327311827 +0000 2024-09-21T15:14:34.509 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-21 15:11:02.327311827 +0000 2024-09-21T15:14:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-21T15:14:34.509 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T15:14:34.564 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-21T15:14:34.564 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-21T15:14:34.564 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000400745 s, 1.3 MB/s 2024-09-21T15:14:34.566 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T15:14:34.573 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 382 Links: 1 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-21 15:11:02.975296879 +0000 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-21 15:11:02.755301954 +0000 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-21 15:11:02.755301954 +0000 2024-09-21T15:14:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-21T15:14:34.578 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T15:14:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-21T15:14:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-21T15:14:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000380689 s, 1.3 MB/s 2024-09-21T15:14:34.631 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T15:14:34.640 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 387 Links: 1 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-21 15:11:03.427286454 +0000 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-21 15:11:03.207291529 +0000 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-21 15:11:03.207291529 +0000 2024-09-21T15:14:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-21T15:14:34.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T15:14:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-21T15:14:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-21T15:14:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000369272 s, 1.4 MB/s 2024-09-21T15:14:34.660 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T15:14:34.710 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:34.710 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-21T15:14:34.710 INFO:tasks.ceph:Generating config... 2024-09-21T15:14:34.712 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-21T15:14:34.712 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-21T15:14:34.712 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-21T15:14:34.712 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T15:14:34.712 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T15:14:34.712 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T15:14:34.712 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T15:14:34.712 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T15:14:34.712 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T15:14:34.712 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-21T15:14:34.712 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-21T15:14:34.712 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T15:14:34.712 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-21T15:14:34.712 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-21T15:14:34.713 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T15:14:34.713 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T15:14:34.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T15:14:34.753 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T15:14:34.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T15:14:34.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T15:14:34.868 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.c', '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]'), ('mon.b', '172.21.15.142')] 2024-09-21T15:14:34.868 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.142', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-21T15:14:34.869 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:34.869 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T15:14:34.875 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --addv c '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]' --add b 172.21.15.142 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:34.967 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:34.967 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid fa0b2d62-c633-4668-b5db-f1f6328d1af9 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = pacific 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:fsid fa0b2d62-c633-4668-b5db-f1f6328d1af9 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-09-21T15:14:34.970194+0000 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-09-21T15:14:34.970194+0000 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 16 (pacific) 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-09-21T15:14:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-21T15:14:34.969 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T15:14:34.974 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fa0b2d62-c633-4668-b5db-f1f6328d1af9... 2024-09-21T15:14:34.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T15:14:34.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T15:14:34.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T15:14:35.007 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-09-21T15:14:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-09-21T15:14:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T15:14:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-09-21T15:14:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-09-21T15:14:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-09-21T15:14:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-09-21T15:14:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-09-21T15:14:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-09-21T15:14:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-09-21T15:14:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T15:14:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-09-21T15:14:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T15:14:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T15:14:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-09-21T15:14:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-09-21T15:14:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-09-21T15:14:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-09-21T15:14:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-09-21T15:14:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-09-21T15:14:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-09-21T15:14:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-09-21T15:14:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T15:14:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T15:14:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-09-21T15:14:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T15:14:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-09-21T15:14:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-09-21T15:14:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-09-21T15:14:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-09-21T15:14:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-09-21T15:14:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.142" 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on pool no app = False 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 5000 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = fa0b2d62-c633-4668-b5db-f1f6328d1af9 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-09-21T15:14:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-09-21T15:14:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-09-21T15:14:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-09-21T15:14:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-09-21T15:14:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-09-21T15:14:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-09-21T15:14:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-09-21T15:14:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-09-21T15:14:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-09-21T15:14:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-09-21T15:14:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-09-21T15:14:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-09-21T15:14:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-09-21T15:14:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression algorithm = zlib 2024-09-21T15:14:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore compression mode = aggressive 2024-09-21T15:14:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-09-21T15:14:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-09-21T15:14:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-09-21T15:14:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-21T15:14:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-09-21T15:14:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-09-21T15:14:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T15:14:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-09-21T15:14:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-09-21T15:14:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: osd blocked scrub grace period = 3600 2024-09-21T15:14:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-09-21T15:14:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: osd max markdown count = 1000 2024-09-21T15:14:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-09-21T15:14:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.021 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:14:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:14:35.023 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-09-21T15:14:35.024 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-09-21T15:14:35.026 INFO:teuthology.orchestra.run.smithi142.stdout:[global] 2024-09-21T15:14:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: chdir = "" 2024-09-21T15:14:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: auth supported = cephx 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: filestore xattr use omap = true 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: mon clock drift allowed = 1.000 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: osd crush chooseleaf type = 0 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: auth debug = true 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on old message = true 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on bug = true 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T15:14:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: mon pg warn max object skew = 0 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default size = 2 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow primary affinity = true 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow pg remap = true 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on legacy crush tunables = false 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on crush straw calc version zero = false 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on no sortbitwise = false 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on osd down out interval zero = false 2024-09-21T15:14:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on too few osds = false 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: mon_allow_pool_size_one = true 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: osd default data pool replay window = 5 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: mon allow pool delete = true 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: mon cluster log file level = debug 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: debug asserts on shutdown = true 2024-09-21T15:14:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: mon health detail to clog = false 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: mon host = "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.142" 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: mon client directed command retry = 5 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on pool no app = False 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject socket failures = 5000 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: fsid = fa0b2d62-c633-4668-b5db-f1f6328d1af9 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout:[osd] 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: osd journal size = 100 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub load threshold = 5.0 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub max interval = 600 2024-09-21T15:14:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: osd mclock profile = high_recovery_ops 2024-09-21T15:14:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: osd recover clone overlap = true 2024-09-21T15:14:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: osd recovery max chunk = 1048576 2024-09-21T15:14:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug shutdown = true 2024-09-21T15:14:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug op order = true 2024-09-21T15:14:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify stray on activate = true 2024-09-21T15:14:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: osd open classes on start = true 2024-09-21T15:14:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug pg log writeout = true 2024-09-21T15:14:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: osd deep scrub update digest min age = 30 2024-09-21T15:14:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: osd map max advance = 10 2024-09-21T15:14:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: journal zero on create = true 2024-09-21T15:14:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: filestore ondisk finisher threads = 3 2024-09-21T15:14:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: filestore apply finisher threads = 3 2024-09-21T15:14:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: bdev debug aio = true 2024-09-21T15:14:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug misdirected ops = true 2024-09-21T15:14:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore block size = 96636764160 2024-09-21T15:14:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore compression algorithm = zlib 2024-09-21T15:14:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore compression mode = aggressive 2024-09-21T15:14:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore fsck on mount = True 2024-09-21T15:14:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluefs = 1/20 2024-09-21T15:14:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluestore = 1/20 2024-09-21T15:14:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-09-21T15:14:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: debug osd = 20 2024-09-21T15:14:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: debug rocksdb = 4/10 2024-09-21T15:14:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T15:14:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd full ratio = 0.9 2024-09-21T15:14:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd nearfull ratio = 0.8 2024-09-21T15:14:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: osd blocked scrub grace period = 3600 2024-09-21T15:14:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: osd failsafe full ratio = 0.95 2024-09-21T15:14:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: osd max markdown count = 1000 2024-09-21T15:14:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: osd objectstore = bluestore 2024-09-21T15:14:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.037 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr] 2024-09-21T15:14:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-09-21T15:14:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: debug mgr = 20 2024-09-21T15:14:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2024-09-21T15:14:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2024-09-21T15:14:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:14:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:14:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: mgr/telemetry/nag = false 2024-09-21T15:14:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.038 INFO:teuthology.orchestra.run.smithi142.stdout:[mon] 2024-09-21T15:14:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-09-21T15:14:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2024-09-21T15:14:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: debug paxos = 20 2024-09-21T15:14:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2024-09-21T15:14:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: mon data avail warn = 5 2024-09-21T15:14:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: mon mgr mkfs grace = 240 2024-09-21T15:14:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:14:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd reporter subtree level = osd 2024-09-21T15:14:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd prime pg temp = true 2024-09-21T15:14:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T15:14:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: # 1m isn't quite enough 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: mon_down_mkfs_grace = 2m 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_filestore_osds = false 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout:[client] 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: rgw cache enabled = true 2024-09-21T15:14:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable ops log = true 2024-09-21T15:14:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable usage log = true 2024-09-21T15:14:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T15:14:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T15:14:35.042 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.a] 2024-09-21T15:14:35.042 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.c] 2024-09-21T15:14:35.042 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.b] 2024-09-21T15:14:35.042 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-09-21T15:14:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-09-21T15:14:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T15:14:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-09-21T15:14:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-09-21T15:14:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-09-21T15:14:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-09-21T15:14:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-09-21T15:14:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-09-21T15:14:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-09-21T15:14:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T15:14:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-09-21T15:14:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T15:14:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T15:14:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-09-21T15:14:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-09-21T15:14:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-09-21T15:14:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-09-21T15:14:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-09-21T15:14:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-09-21T15:14:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-09-21T15:14:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-09-21T15:14:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T15:14:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T15:14:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-09-21T15:14:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T15:14:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-09-21T15:14:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-09-21T15:14:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-09-21T15:14:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-09-21T15:14:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-09-21T15:14:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.142" 2024-09-21T15:14:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2024-09-21T15:14:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2024-09-21T15:14:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 5000 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = fa0b2d62-c633-4668-b5db-f1f6328d1af9 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-09-21T15:14:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-09-21T15:14:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-09-21T15:14:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-09-21T15:14:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-09-21T15:14:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-09-21T15:14:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-09-21T15:14:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-09-21T15:14:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-09-21T15:14:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-09-21T15:14:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-09-21T15:14:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-09-21T15:14:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-09-21T15:14:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-09-21T15:14:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-09-21T15:14:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-09-21T15:14:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-09-21T15:14:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore compression algorithm = zlib 2024-09-21T15:14:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore compression mode = aggressive 2024-09-21T15:14:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-09-21T15:14:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-09-21T15:14:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-09-21T15:14:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-21T15:14:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-09-21T15:14:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-09-21T15:14:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T15:14:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-09-21T15:14:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-09-21T15:14:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: osd blocked scrub grace period = 3600 2024-09-21T15:14:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-09-21T15:14:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: osd max markdown count = 1000 2024-09-21T15:14:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-21T15:14:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T15:14:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T15:14:35.059 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-09-21T15:14:35.060 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-09-21T15:14:35.060 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-09-21T15:14:35.060 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T15:14:35.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T15:14:35.126 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T15:14:35.127 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:35.127 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T15:14:35.133 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:35.133 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T15:14:35.180 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:14:35.180 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:35.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T15:14:35.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T15:14:35.207 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:35.207 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:35.214 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:14:35.214 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:35.214 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T15:14:35.214 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T15:14:35.241 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:35.241 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:35.247 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-09-21T15:14:35.247 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:14:35.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T15:14:35.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T15:14:35.274 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:14:35.274 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:35.324 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T15:14:35.324 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T15:14:35.325 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T15:14:35.375 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T15:14:35.392 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T15:14:35.392 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T15:14:35.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T15:14:35.433 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T15:14:35.457 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T15:14:35.457 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-21T15:14:35.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T15:14:35.473 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:35.473 INFO:tasks.ceph:role: osd.0 2024-09-21T15:14:35.473 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:14:35.474 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T15:14:36.005 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:14:36.006 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:14:36.008 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-21T15:14:36.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T15:14:36.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T15:14:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:14:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T15:14:36.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T15:14:36.178 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:36.178 INFO:tasks.ceph:role: osd.1 2024-09-21T15:14:36.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:14:36.178 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:14:36.693 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:14:36.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-21T15:14:36.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T15:14:36.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T15:14:36.789 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T15:14:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:14:36.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T15:14:36.801 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:36.802 INFO:tasks.ceph:role: osd.2 2024-09-21T15:14:36.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:14:36.802 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:14:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:14:37.295 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-09-21T15:14:37.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T15:14:37.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T15:14:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T15:14:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:14:37.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:37.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:37.579+0000 7f63b7ae1880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-21T15:14:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:37.579+0000 7f63b7ae1880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-21T15:14:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:37.579+0000 7f63b7ae1880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-21T15:14:37.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:37.579+0000 7f63b7ae1880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-21T15:14:37.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:37.579+0000 7f63b7ae1880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-21T15:14:37.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:37.587+0000 7f63b7ae1880 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-21T15:14:40.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T15:14:40.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:40.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:40.647+0000 7f36eb3ae880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-21T15:14:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:40.647+0000 7f36eb3ae880 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-21T15:14:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:40.647+0000 7f36eb3ae880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-21T15:14:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:40.647+0000 7f36eb3ae880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-21T15:14:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:40.647+0000 7f36eb3ae880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-21T15:14:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:40.647+0000 7f36eb3ae880 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-21T15:14:43.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T15:14:43.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:43.747+0000 7fd8ddca7880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-21T15:14:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:43.747+0000 7fd8ddca7880 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-21T15:14:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:43.747+0000 7fd8ddca7880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-21T15:14:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:43.747+0000 7fd8ddca7880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-21T15:14:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:43.747+0000 7fd8ddca7880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-21T15:14:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-21T15:14:43.751+0000 7fd8ddca7880 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-21T15:14:46.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T15:14:46.770 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-21T15:14:46.787 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:46.787 INFO:tasks.ceph:role: osd.3 2024-09-21T15:14:46.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:14:46.787 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T15:14:47.328 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:14:47.329 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:14:47.330 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-09-21T15:14:47.330 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-21T15:14:47.422 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-21T15:14:47.434 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T15:14:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:14:47.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-21T15:14:47.493 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:47.493 INFO:tasks.ceph:role: osd.4 2024-09-21T15:14:47.493 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:14:47.494 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T15:14:48.008 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:14:48.009 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:14:48.011 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-09-21T15:14:48.011 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-21T15:14:48.043 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-21T15:14:48.052 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T15:14:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:14:48.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-21T15:14:48.109 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T15:14:48.109 INFO:tasks.ceph:role: osd.5 2024-09-21T15:14:48.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:14:48.109 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T15:14:48.588 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:14:48.588 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:14:48.589 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:14:48.590 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-09-21T15:14:48.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-21T15:14:48.643 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-21T15:14:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:14:48.655 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T15:14:48.656 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:48.734+0000 7fcadf321880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-21T15:14:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:48.738+0000 7fcadf321880 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-21T15:14:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:48.738+0000 7fcadf321880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-21T15:14:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:48.738+0000 7fcadf321880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-21T15:14:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:48.738+0000 7fcadf321880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-21T15:14:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:48.742+0000 7fcadf321880 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-21T15:14:51.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T15:14:51.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:51.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:51.813+0000 7f8a4206c880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-21T15:14:51.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:51.813+0000 7f8a4206c880 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-21T15:14:51.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:51.813+0000 7f8a4206c880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-21T15:14:51.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:51.813+0000 7f8a4206c880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-21T15:14:51.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:51.813+0000 7f8a4206c880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-21T15:14:51.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:51.813+0000 7f8a4206c880 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-21T15:14:54.821 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T15:14:54.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:54.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:54.913+0000 7f0f50ae8880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-21T15:14:54.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:54.913+0000 7f0f50ae8880 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-21T15:14:54.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:54.913+0000 7f0f50ae8880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-21T15:14:54.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:54.913+0000 7f0f50ae8880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-21T15:14:54.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:54.913+0000 7f0f50ae8880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-21T15:14:54.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-21T15:14:54.913+0000 7f0f50ae8880 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-21T15:14:57.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T15:14:57.946 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T15:14:57.946 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:57.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T15:14:57.963 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:57.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T15:14:57.977 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:14:57.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T15:14:58.031 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:58.031 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T15:14:58.047 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:58.047 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-21T15:14:58.063 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:58.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-21T15:14:58.121 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:14:58.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-21T15:14:58.136 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:14:58.136 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T15:14:58.143 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T15:14:58.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T15:14:58.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T15:14:58.156 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.0] 2024-09-21T15:14:58.156 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDd4u5mLPq+IhAAZ61mbLBHPpdBmBS3MW72+g== 2024-09-21T15:14:58.156 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.1] 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDg4u5mvs++JhAAsXyI25O02WqHF30AuD3zag== 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.2] 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDj4u5mNz7JLBAAU9EPWVHLxOAJyCpCDPk1mw== 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.x] 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDb4u5moTVXFhAAHRRJMlp/v3pa33D6ObpHGQ== 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.3] 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDo4u5mes4yLBAAFJDWuw1LtOrcUEb0VklamA== 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.4] 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDr4u5mJ8HYMBAARb/y4HgJqSuNzyP8wCkKBQ== 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.5] 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDu4u5mpyrXNhAAzsvD67En+xH50OPhqnNOtg== 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout:[client.0] 2024-09-21T15:14:58.157 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQDb4u5mESvPGRAAmg4jRXKzI1SzImV5w1KYSQ== 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDd4u5mLPq+IhAAZ61mbLBHPpdBmBS3MW72+g== 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDg4u5mvs++JhAAsXyI25O02WqHF30AuD3zag== 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDj4u5mNz7JLBAAU9EPWVHLxOAJyCpCDPk1mw== 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDb4u5moTVXFhAAHRRJMlp/v3pa33D6ObpHGQ== 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDo4u5mes4yLBAAFJDWuw1LtOrcUEb0VklamA== 2024-09-21T15:14:58.159 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-09-21T15:14:58.160 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDr4u5mJ8HYMBAARb/y4HgJqSuNzyP8wCkKBQ== 2024-09-21T15:14:58.160 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-09-21T15:14:58.160 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDu4u5mpyrXNhAAzsvD67En+xH50OPhqnNOtg== 2024-09-21T15:14:58.160 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-09-21T15:14:58.160 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDb4u5mESvPGRAAmg4jRXKzI1SzImV5w1KYSQ== 2024-09-21T15:14:58.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.256 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.307 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T15:14:58.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T15:14:58.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.408 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.513 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:14:58.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T15:14:58.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T15:14:58.619 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T15:14:58.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T15:14:58.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T15:14:58.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T15:14:58.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-21T15:14:58.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T15:14:58.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-21T15:14:58.904 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-21T15:14:58.921 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T15:14:59.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-21T15:14:59.015 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:59.017 DEBUG:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:14:59.021 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T15:14:59.021 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T15:14:59.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T15:14:59.063 INFO:tasks.ceph.mon.a:Started 2024-09-21T15:14:59.064 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-21T15:14:59.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-21T15:14:59.066 INFO:tasks.ceph.mon.c:Started 2024-09-21T15:14:59.066 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-21T15:14:59.066 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-21T15:14:59.068 INFO:tasks.ceph.mon.b:Started 2024-09-21T15:14:59.068 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T15:14:59.068 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T15:14:59.069 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T15:14:59.071 INFO:tasks.ceph.mgr.x:Started 2024-09-21T15:14:59.071 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T15:14:59.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T15:15:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-09-21T15:15:04.392 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T15:15:04.392 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T15:15:04.393 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:15:04.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T15:15:04.404 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:15:04.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T15:15:04.413 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-21T15:15:04.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T15:15:04.422 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:15:04.422 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-21T15:15:04.439 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:15:04.439 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-21T15:15:04.482 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:04.477+0000 7fac9b4e82c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T15:15:04.497 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-21T15:15:04.497 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-21T15:15:04.509 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 6194913c-995a-4f8b-b2e4-1ae24c7fc1c6 0 2024-09-21T15:15:04.711 INFO:teuthology.orchestra.run.smithi142.stdout:0 2024-09-21T15:15:04.725 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 496a596a-f018-4010-bd93-6df3cfbef67f 1 2024-09-21T15:15:04.927 INFO:teuthology.orchestra.run.smithi142.stdout:1 2024-09-21T15:15:04.940 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 0131c1a7-80b4-448b-b7b2-89e6ba9ec584 2 2024-09-21T15:15:05.149 INFO:teuthology.orchestra.run.smithi142.stdout:2 2024-09-21T15:15:05.162 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new edacb78d-2a42-4dd8-a648-86031fd9ad18 3 2024-09-21T15:15:05.225 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:05.217+0000 7fac9b4e82c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T15:15:05.283 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:05.277+0000 7fac9b4e82c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T15:15:05.388 INFO:teuthology.orchestra.run.smithi142.stdout:3 2024-09-21T15:15:05.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new bd5f5e45-2c13-432b-9aba-69c731ca2248 4 2024-09-21T15:15:05.481 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:05.473+0000 7fac9b4e82c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T15:15:05.630 INFO:teuthology.orchestra.run.smithi142.stdout:4 2024-09-21T15:15:05.643 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 52ceefef-20a8-41fb-ab77-053021d17105 5 2024-09-21T15:15:05.734 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:05.733+0000 7fac9b4e82c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T15:15:05.791 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:05.789+0000 7fac9b4e82c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T15:15:05.848 INFO:teuthology.orchestra.run.smithi142.stdout:5 2024-09-21T15:15:05.864 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T15:15:05.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T15:15:05.866 INFO:tasks.ceph.osd.0:Started 2024-09-21T15:15:05.866 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T15:15:05.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-21T15:15:05.868 INFO:tasks.ceph.osd.1:Started 2024-09-21T15:15:05.868 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T15:15:05.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T15:15:05.871 INFO:tasks.ceph.osd.2:Started 2024-09-21T15:15:05.871 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-21T15:15:05.871 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-21T15:15:05.873 INFO:tasks.ceph.osd.3:Started 2024-09-21T15:15:05.874 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-21T15:15:05.874 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-21T15:15:05.876 INFO:tasks.ceph.osd.4:Started 2024-09-21T15:15:05.876 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-21T15:15:05.876 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-21T15:15:05.878 INFO:tasks.ceph.osd.5:Started 2024-09-21T15:15:05.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:15:05.995 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:05.993+0000 7fac9b4e82c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T15:15:06.057 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:06.053+0000 7fac9b4e82c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T15:15:06.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:06.070 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:15:05.841121+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T15:15:05.841121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:15:06.083 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T15:15:06.083 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T15:15:06.121 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:06.117+0000 7fac9b4e82c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T15:15:06.284 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:06.283+0000 7fcfff608880 -1 Falling back to public interface 2024-09-21T15:15:06.500 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:06.499+0000 7f5b89a01880 -1 Falling back to public interface 2024-09-21T15:15:06.500 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:06.497+0000 7fa01aa92880 -1 Falling back to public interface 2024-09-21T15:15:06.504 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:06.501+0000 7fb974557880 -1 Falling back to public interface 2024-09-21T15:15:06.511 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:06.509+0000 7f926d0d2880 -1 Falling back to public interface 2024-09-21T15:15:06.517 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:06.513+0000 7fac9b4e82c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T15:15:06.524 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:06.523+0000 7fe68a52c880 -1 Falling back to public interface 2024-09-21T15:15:06.635 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:06.633+0000 7fac9b4e82c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T15:15:06.689 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T15:15:06.884 INFO:teuthology.misc.health.smithi067.stdout: 2024-09-21T15:15:06.884 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":8,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:15:05.841121+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T15:15:05.841121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:15:06.895 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-21T15:15:06.969 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:06.965+0000 7fac9b4e82c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T15:15:07.153 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:07.149+0000 7fac9b4e82c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T15:15:07.221 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:07.217+0000 7fac9b4e82c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T15:15:07.384 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:07.381+0000 7fac9b4e82c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T15:15:08.417 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:08.413+0000 7fac9b4e82c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T15:15:08.487 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:08.485+0000 7fac9b4e82c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T15:15:08.639 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:08.637+0000 7fac9b4e82c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T15:15:08.710 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:08.705+0000 7fac9b4e82c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T15:15:08.786 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:08.781+0000 7fac9b4e82c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T15:15:09.126 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:09.121+0000 7fac9b4e82c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T15:15:09.201 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:09.197+0000 7fac9b4e82c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T15:15:09.626 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:09.621+0000 7f926d0d2880 -1 osd.4 0 log_to_monitors true 2024-09-21T15:15:09.676 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:09.674+0000 7fcfff608880 -1 osd.2 0 log_to_monitors true 2024-09-21T15:15:09.818 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:09.813+0000 7fa01aa92880 -1 osd.5 0 log_to_monitors true 2024-09-21T15:15:09.881 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:09.877+0000 7fac9b4e82c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T15:15:10.351 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:10.350+0000 7f5b89a01880 -1 osd.1 0 log_to_monitors true 2024-09-21T15:15:10.352 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:10.350+0000 7fe68a52c880 -1 osd.0 0 log_to_monitors true 2024-09-21T15:15:10.353 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:10.349+0000 7fb974557880 -1 osd.3 0 log_to_monitors true 2024-09-21T15:15:10.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:10.921+0000 7fac8205d700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T15:15:10.925 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:15:10.921+0000 7fac8205d700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T15:15:13.501 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T15:15:13.721 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:13.717+0000 7f9268f11700 -1 osd.4 0 waiting for initial osdmap 2024-09-21T15:15:13.726 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:13.721+0000 7f92609c0700 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:15:13.875 INFO:teuthology.misc.health.smithi067.stdout: 2024-09-21T15:15:13.875 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:15:11.703314+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T15:15:05.841121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:15:13.890 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-21T15:15:13.930 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:13.930+0000 7fcffb447700 -1 osd.2 0 waiting for initial osdmap 2024-09-21T15:15:13.935 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:13.934+0000 7fcff2ef6700 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:15:14.928 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:14.925+0000 7fb970396700 -1 osd.3 0 waiting for initial osdmap 2024-09-21T15:15:14.935 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:14.933+0000 7fb967e45700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:15:15.016 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:15.013+0000 7fa017258700 -1 osd.5 0 waiting for initial osdmap 2024-09-21T15:15:15.025 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:15.021+0000 7fa00e380700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:15:15.106 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:15.106+0000 7f5b85840700 -1 osd.1 0 waiting for initial osdmap 2024-09-21T15:15:15.112 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:15.110+0000 7f5b7d2ef700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:15:15.113 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:15.110+0000 7fe68636b700 -1 osd.0 0 waiting for initial osdmap 2024-09-21T15:15:15.118 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:15.118+0000 7fe67de1a700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:15:20.497 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T15:15:20.797 INFO:teuthology.misc.health.smithi067.stdout: 2024-09-21T15:15:20.797 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:15:18.774506+0000","last_up_change":"2024-09-21T15:15:15.737314+0000","last_in_change":"2024-09-21T15:15:05.841121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:15:15.995966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6817","nonce":3600997419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6819","nonce":3600997419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6823","nonce":3600997419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6821","nonce":3600997419}]},"public_addr":"172.21.15.67:6817/3600997419","cluster_addr":"172.21.15.67:6819/3600997419","heartbeat_back_addr":"172.21.15.67:6823/3600997419","heartbeat_front_addr":"172.21.15.67:6821/3600997419","state":["exists","up"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6809","nonce":1817390160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6811","nonce":1817390160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6815","nonce":1817390160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6813","nonce":1817390160}]},"public_addr":"172.21.15.67:6809/1817390160","cluster_addr":"172.21.15.67:6811/1817390160","heartbeat_back_addr":"172.21.15.67:6815/1817390160","heartbeat_front_addr":"172.21.15.67:6813/1817390160","state":["exists","up"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6801","nonce":3371454484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6803","nonce":3371454484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6807","nonce":3371454484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6805","nonce":3371454484}]},"public_addr":"172.21.15.67:6801/3371454484","cluster_addr":"172.21.15.67:6803/3371454484","heartbeat_back_addr":"172.21.15.67:6807/3371454484","heartbeat_front_addr":"172.21.15.67:6805/3371454484","state":["exists","up"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6809","nonce":1069930087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6811","nonce":1069930087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6815","nonce":1069930087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6813","nonce":1069930087}]},"public_addr":"172.21.15.142:6809/1069930087","cluster_addr":"172.21.15.142:6811/1069930087","heartbeat_back_addr":"172.21.15.142:6815/1069930087","heartbeat_front_addr":"172.21.15.142:6813/1069930087","state":["exists","up"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6817","nonce":3606282773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6819","nonce":3606282773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6823","nonce":3606282773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6821","nonce":3606282773}]},"public_addr":"172.21.15.142:6817/3606282773","cluster_addr":"172.21.15.142:6819/3606282773","heartbeat_back_addr":"172.21.15.142:6823/3606282773","heartbeat_front_addr":"172.21.15.142:6821/3606282773","state":["exists","up"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6801","nonce":3327577487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6803","nonce":3327577487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6807","nonce":3327577487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6805","nonce":3327577487}]},"public_addr":"172.21.15.142:6801/3327577487","cluster_addr":"172.21.15.142:6803/3327577487","heartbeat_back_addr":"172.21.15.142:6807/3327577487","heartbeat_front_addr":"172.21.15.142:6805/3327577487","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.386986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.358863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.645118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.318690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.618213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.843287+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:15:20.812 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-21T15:15:20.812 INFO:tasks.ceph:Creating RBD pool 2024-09-21T15:15:20.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T15:15:21.810 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'rbd' created 2024-09-21T15:15:21.822 DEBUG:teuthology.orchestra.run.smithi067:> rbd --cluster ceph pool init rbd 2024-09-21T15:15:24.837 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T15:15:24.838 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T15:15:24.838 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T15:15:24.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T15:15:24.838 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T15:15:25.192 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:25.205 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1484190093},{"type":"v1","addr":"172.21.15.142:6825","nonce":1484190093}]},"active_addr":"172.21.15.142:6825/1484190093","active_change":"2024-09-21T15:15:09.904419+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":3289515266}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":1169731978}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":64492699}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.142:0","nonce":1519453312}]}]} 2024-09-21T15:15:25.207 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T15:15:25.207 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T15:15:25.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:15:25.513 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:25.514 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:15:24.821776+0000","last_up_change":"2024-09-21T15:15:15.737314+0000","last_in_change":"2024-09-21T15:15:05.841121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:15:15.995966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:15:21.158896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6817","nonce":3600997419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6819","nonce":3600997419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6823","nonce":3600997419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6821","nonce":3600997419}]},"public_addr":"172.21.15.67:6817/3600997419","cluster_addr":"172.21.15.67:6819/3600997419","heartbeat_back_addr":"172.21.15.67:6823/3600997419","heartbeat_front_addr":"172.21.15.67:6821/3600997419","state":["exists","up"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6809","nonce":1817390160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6811","nonce":1817390160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6815","nonce":1817390160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6813","nonce":1817390160}]},"public_addr":"172.21.15.67:6809/1817390160","cluster_addr":"172.21.15.67:6811/1817390160","heartbeat_back_addr":"172.21.15.67:6815/1817390160","heartbeat_front_addr":"172.21.15.67:6813/1817390160","state":["exists","up"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6801","nonce":3371454484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6803","nonce":3371454484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6807","nonce":3371454484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6805","nonce":3371454484}]},"public_addr":"172.21.15.67:6801/3371454484","cluster_addr":"172.21.15.67:6803/3371454484","heartbeat_back_addr":"172.21.15.67:6807/3371454484","heartbeat_front_addr":"172.21.15.67:6805/3371454484","state":["exists","up"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6809","nonce":1069930087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6811","nonce":1069930087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6815","nonce":1069930087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6813","nonce":1069930087}]},"public_addr":"172.21.15.142:6809/1069930087","cluster_addr":"172.21.15.142:6811/1069930087","heartbeat_back_addr":"172.21.15.142:6815/1069930087","heartbeat_front_addr":"172.21.15.142:6813/1069930087","state":["exists","up"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6817","nonce":3606282773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6819","nonce":3606282773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6823","nonce":3606282773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6821","nonce":3606282773}]},"public_addr":"172.21.15.142:6817/3606282773","cluster_addr":"172.21.15.142:6819/3606282773","heartbeat_back_addr":"172.21.15.142:6823/3606282773","heartbeat_front_addr":"172.21.15.142:6821/3606282773","state":["exists","up"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6801","nonce":3327577487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6803","nonce":3327577487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6807","nonce":3327577487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6805","nonce":3327577487}]},"public_addr":"172.21.15.142:6801/3327577487","cluster_addr":"172.21.15.142:6803/3327577487","heartbeat_back_addr":"172.21.15.142:6807/3327577487","heartbeat_front_addr":"172.21.15.142:6805/3327577487","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.386986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.358863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.645118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.318690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.618213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.843287+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:15:25.528 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T15:15:25.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:15:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:15:24.821776+0000","last_up_change":"2024-09-21T15:15:15.737314+0000","last_in_change":"2024-09-21T15:15:05.841121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:15:15.995966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:15:21.158896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6817","nonce":3600997419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6819","nonce":3600997419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6823","nonce":3600997419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6821","nonce":3600997419}]},"public_addr":"172.21.15.67:6817/3600997419","cluster_addr":"172.21.15.67:6819/3600997419","heartbeat_back_addr":"172.21.15.67:6823/3600997419","heartbeat_front_addr":"172.21.15.67:6821/3600997419","state":["exists","up"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6809","nonce":1817390160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6811","nonce":1817390160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6815","nonce":1817390160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6813","nonce":1817390160}]},"public_addr":"172.21.15.67:6809/1817390160","cluster_addr":"172.21.15.67:6811/1817390160","heartbeat_back_addr":"172.21.15.67:6815/1817390160","heartbeat_front_addr":"172.21.15.67:6813/1817390160","state":["exists","up"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6801","nonce":3371454484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6803","nonce":3371454484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6807","nonce":3371454484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6805","nonce":3371454484}]},"public_addr":"172.21.15.67:6801/3371454484","cluster_addr":"172.21.15.67:6803/3371454484","heartbeat_back_addr":"172.21.15.67:6807/3371454484","heartbeat_front_addr":"172.21.15.67:6805/3371454484","state":["exists","up"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6809","nonce":1069930087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6811","nonce":1069930087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6815","nonce":1069930087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6813","nonce":1069930087}]},"public_addr":"172.21.15.142:6809/1069930087","cluster_addr":"172.21.15.142:6811/1069930087","heartbeat_back_addr":"172.21.15.142:6815/1069930087","heartbeat_front_addr":"172.21.15.142:6813/1069930087","state":["exists","up"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6817","nonce":3606282773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6819","nonce":3606282773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6823","nonce":3606282773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6821","nonce":3606282773}]},"public_addr":"172.21.15.142:6817/3606282773","cluster_addr":"172.21.15.142:6819/3606282773","heartbeat_back_addr":"172.21.15.142:6823/3606282773","heartbeat_front_addr":"172.21.15.142:6821/3606282773","state":["exists","up"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6801","nonce":3327577487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6803","nonce":3327577487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6807","nonce":3327577487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6805","nonce":3327577487}]},"public_addr":"172.21.15.142:6801/3327577487","cluster_addr":"172.21.15.142:6803/3327577487","heartbeat_back_addr":"172.21.15.142:6807/3327577487","heartbeat_front_addr":"172.21.15.142:6805/3327577487","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.386986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.358863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.645118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.318690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.618213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.843287+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:15:25.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T15:15:25.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T15:15:25.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T15:15:25.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T15:15:25.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T15:15:25.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T15:15:26.013 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-09-21T15:15:26.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:15:26.032 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-09-21T15:15:26.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:15:26.059 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-09-21T15:15:26.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:15:26.063 INFO:teuthology.orchestra.run.smithi067.stdout:51539607557 2024-09-21T15:15:26.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:15:26.067 INFO:teuthology.orchestra.run.smithi067.stdout:51539607559 2024-09-21T15:15:26.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:15:26.068 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-09-21T15:15:26.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:15:26.395 INFO:teuthology.orchestra.run.smithi067.stdout:55834574851 2024-09-21T15:15:26.410 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2024-09-21T15:15:26.444 INFO:teuthology.orchestra.run.smithi067.stdout:51539607555 2024-09-21T15:15:26.451 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-09-21T15:15:26.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.2 2024-09-21T15:15:26.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-09-21T15:15:26.479 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-09-21T15:15:26.495 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2024-09-21T15:15:26.564 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-09-21T15:15:26.576 INFO:teuthology.orchestra.run.smithi067.stdout:51539607557 2024-09-21T15:15:26.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2024-09-21T15:15:26.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607559 got 51539607557 for osd.4 2024-09-21T15:15:27.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:15:27.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:15:27.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:15:27.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:15:27.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:15:27.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:15:27.826 INFO:teuthology.orchestra.run.smithi067.stdout:51539607556 2024-09-21T15:15:27.828 INFO:teuthology.orchestra.run.smithi067.stdout:55834574851 2024-09-21T15:15:27.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.2 2024-09-21T15:15:27.851 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2024-09-21T15:15:27.899 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-09-21T15:15:27.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-09-21T15:15:27.929 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-09-21T15:15:27.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2024-09-21T15:15:27.942 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-09-21T15:15:27.942 INFO:teuthology.orchestra.run.smithi067.stdout:51539607558 2024-09-21T15:15:27.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607559 got 51539607558 for osd.4 2024-09-21T15:15:27.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.5 2024-09-21T15:15:28.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:15:28.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:15:28.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:15:28.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:15:28.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:15:28.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:15:29.199 INFO:teuthology.orchestra.run.smithi067.stdout:51539607557 2024-09-21T15:15:29.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.2 2024-09-21T15:15:29.214 DEBUG:teuthology.parallel:result is None 2024-09-21T15:15:29.245 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-09-21T15:15:29.261 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-09-21T15:15:29.264 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.0 2024-09-21T15:15:29.264 DEBUG:teuthology.parallel:result is None 2024-09-21T15:15:29.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.1 2024-09-21T15:15:29.279 DEBUG:teuthology.parallel:result is None 2024-09-21T15:15:29.349 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-09-21T15:15:29.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.5 2024-09-21T15:15:29.366 DEBUG:teuthology.parallel:result is None 2024-09-21T15:15:29.380 INFO:teuthology.orchestra.run.smithi067.stdout:51539607559 2024-09-21T15:15:29.390 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-09-21T15:15:29.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607559 got 51539607559 for osd.4 2024-09-21T15:15:29.394 DEBUG:teuthology.parallel:result is None 2024-09-21T15:15:29.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.3 2024-09-21T15:15:29.404 DEBUG:teuthology.parallel:result is None 2024-09-21T15:15:29.405 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T15:15:29.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:15:29.701 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:15:29.716 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-21T15:15:29.184356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":187944,"kb_used_data":1224,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566043096,"statfs":{"total":579820584960,"available":579628130304,"internally_reserved":0,"allocated":1253376,"data_stored":1553680,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.373975"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.830605+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:15:24.830605+0000","last_peered":"2024-09-21T15:15:24.830605+0000","last_clean":"2024-09-21T15:15:24.830605+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:15:24.830605+0000","last_undegraded":"2024-09-21T15:15:24.830605+0000","last_fullsized":"2024-09-21T15:15:24.830605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.826574+0000","last_change":"2024-09-21T15:15:24.826574+0000","last_active":"2024-09-21T15:15:24.826574+0000","last_peered":"2024-09-21T15:15:24.826574+0000","last_clean":"2024-09-21T15:15:24.826574+0000","last_became_active":"2024-09-21T15:15:22.824152+0000","last_became_peered":"2024-09-21T15:15:22.824152+0000","last_unstale":"2024-09-21T15:15:24.826574+0000","last_undegraded":"2024-09-21T15:15:24.826574+0000","last_fullsized":"2024-09-21T15:15:24.826574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:54.629041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066240600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829674+0000","last_change":"2024-09-21T15:15:24.829674+0000","last_active":"2024-09-21T15:15:24.829674+0000","last_peered":"2024-09-21T15:15:24.829674+0000","last_clean":"2024-09-21T15:15:24.829674+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:15:24.829674+0000","last_undegraded":"2024-09-21T15:15:24.829674+0000","last_fullsized":"2024-09-21T15:15:24.829674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:48.076212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.427839+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:15:25.427839+0000","last_peered":"2024-09-21T15:15:25.427839+0000","last_clean":"2024-09-21T15:15:25.427839+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:15:25.427839+0000","last_undegraded":"2024-09-21T15:15:25.427839+0000","last_fullsized":"2024-09-21T15:15:25.427839+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.834145+0000","last_change":"2024-09-21T15:15:24.830325+0000","last_active":"2024-09-21T15:15:24.834145+0000","last_peered":"2024-09-21T15:15:24.834145+0000","last_clean":"2024-09-21T15:15:24.834145+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:15:24.834145+0000","last_undegraded":"2024-09-21T15:15:24.834145+0000","last_fullsized":"2024-09-21T15:15:24.834145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:55:21.434241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.511790+0000","last_change":"2024-09-21T15:15:25.511790+0000","last_active":"2024-09-21T15:15:25.511790+0000","last_peered":"2024-09-21T15:15:25.511790+0000","last_clean":"2024-09-21T15:15:25.511790+0000","last_became_active":"2024-09-21T15:15:22.828112+0000","last_became_peered":"2024-09-21T15:15:22.828112+0000","last_unstale":"2024-09-21T15:15:25.511790+0000","last_undegraded":"2024-09-21T15:15:25.511790+0000","last_fullsized":"2024-09-21T15:15:25.511790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:10:48.407777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069731800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.659947+0000","last_change":"2024-09-21T15:15:25.660255+0000","last_active":"2024-09-21T15:15:25.659947+0000","last_peered":"2024-09-21T15:15:25.659947+0000","last_clean":"2024-09-21T15:15:25.659947+0000","last_became_active":"2024-09-21T15:15:22.821654+0000","last_became_peered":"2024-09-21T15:15:22.821654+0000","last_unstale":"2024-09-21T15:15:25.659947+0000","last_undegraded":"2024-09-21T15:15:25.659947+0000","last_fullsized":"2024-09-21T15:15:25.659947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:55.119052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000470638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829906+0000","last_change":"2024-09-21T15:15:24.829906+0000","last_active":"2024-09-21T15:15:24.829906+0000","last_peered":"2024-09-21T15:15:24.829906+0000","last_clean":"2024-09-21T15:15:24.829906+0000","last_became_active":"2024-09-21T15:15:22.819834+0000","last_became_peered":"2024-09-21T15:15:22.819834+0000","last_unstale":"2024-09-21T15:15:24.829906+0000","last_undegraded":"2024-09-21T15:15:24.829906+0000","last_fullsized":"2024-09-21T15:15:24.829906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:32.673395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:15:23.818914+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:15:23.818914+0000","last_peered":"2024-09-21T15:15:23.818914+0000","last_clean":"2024-09-21T15:15:23.818914+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:15:23.818914+0000","last_undegraded":"2024-09-21T15:15:23.818914+0000","last_fullsized":"2024-09-21T15:15:23.818914+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31312,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340528,"statfs":{"total":96636764160,"available":96604700672,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:15:29.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:15:30.019 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:30.023 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:15:30.034 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T15:15:29.917954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":187944,"kb_used_data":1224,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566043096,"statfs":{"total":579820584960,"available":579628130304,"internally_reserved":0,"allocated":1253376,"data_stored":1553680,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.104985"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.830605+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:15:24.830605+0000","last_peered":"2024-09-21T15:15:24.830605+0000","last_clean":"2024-09-21T15:15:24.830605+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:15:24.830605+0000","last_undegraded":"2024-09-21T15:15:24.830605+0000","last_fullsized":"2024-09-21T15:15:24.830605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.826574+0000","last_change":"2024-09-21T15:15:24.826574+0000","last_active":"2024-09-21T15:15:24.826574+0000","last_peered":"2024-09-21T15:15:24.826574+0000","last_clean":"2024-09-21T15:15:24.826574+0000","last_became_active":"2024-09-21T15:15:22.824152+0000","last_became_peered":"2024-09-21T15:15:22.824152+0000","last_unstale":"2024-09-21T15:15:24.826574+0000","last_undegraded":"2024-09-21T15:15:24.826574+0000","last_fullsized":"2024-09-21T15:15:24.826574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:54.629041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066240600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829674+0000","last_change":"2024-09-21T15:15:24.829674+0000","last_active":"2024-09-21T15:15:24.829674+0000","last_peered":"2024-09-21T15:15:24.829674+0000","last_clean":"2024-09-21T15:15:24.829674+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:15:24.829674+0000","last_undegraded":"2024-09-21T15:15:24.829674+0000","last_fullsized":"2024-09-21T15:15:24.829674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:48.076212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.427839+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:15:25.427839+0000","last_peered":"2024-09-21T15:15:25.427839+0000","last_clean":"2024-09-21T15:15:25.427839+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:15:25.427839+0000","last_undegraded":"2024-09-21T15:15:25.427839+0000","last_fullsized":"2024-09-21T15:15:25.427839+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.834145+0000","last_change":"2024-09-21T15:15:24.830325+0000","last_active":"2024-09-21T15:15:24.834145+0000","last_peered":"2024-09-21T15:15:24.834145+0000","last_clean":"2024-09-21T15:15:24.834145+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:15:24.834145+0000","last_undegraded":"2024-09-21T15:15:24.834145+0000","last_fullsized":"2024-09-21T15:15:24.834145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:55:21.434241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.511790+0000","last_change":"2024-09-21T15:15:25.511790+0000","last_active":"2024-09-21T15:15:25.511790+0000","last_peered":"2024-09-21T15:15:25.511790+0000","last_clean":"2024-09-21T15:15:25.511790+0000","last_became_active":"2024-09-21T15:15:22.828112+0000","last_became_peered":"2024-09-21T15:15:22.828112+0000","last_unstale":"2024-09-21T15:15:25.511790+0000","last_undegraded":"2024-09-21T15:15:25.511790+0000","last_fullsized":"2024-09-21T15:15:25.511790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:10:48.407777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069731800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.659947+0000","last_change":"2024-09-21T15:15:25.660255+0000","last_active":"2024-09-21T15:15:25.659947+0000","last_peered":"2024-09-21T15:15:25.659947+0000","last_clean":"2024-09-21T15:15:25.659947+0000","last_became_active":"2024-09-21T15:15:22.821654+0000","last_became_peered":"2024-09-21T15:15:22.821654+0000","last_unstale":"2024-09-21T15:15:25.659947+0000","last_undegraded":"2024-09-21T15:15:25.659947+0000","last_fullsized":"2024-09-21T15:15:25.659947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:55.119052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000470638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829906+0000","last_change":"2024-09-21T15:15:24.829906+0000","last_active":"2024-09-21T15:15:24.829906+0000","last_peered":"2024-09-21T15:15:24.829906+0000","last_clean":"2024-09-21T15:15:24.829906+0000","last_became_active":"2024-09-21T15:15:22.819834+0000","last_became_peered":"2024-09-21T15:15:22.819834+0000","last_unstale":"2024-09-21T15:15:24.829906+0000","last_undegraded":"2024-09-21T15:15:24.829906+0000","last_fullsized":"2024-09-21T15:15:24.829906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:32.673395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:15:23.818914+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:15:23.818914+0000","last_peered":"2024-09-21T15:15:23.818914+0000","last_clean":"2024-09-21T15:15:23.818914+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:15:23.818914+0000","last_undegraded":"2024-09-21T15:15:23.818914+0000","last_fullsized":"2024-09-21T15:15:23.818914+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31312,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340528,"statfs":{"total":96636764160,"available":96604700672,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:15:30.035 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T15:15:30.036 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T15:15:30.036 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T15:15:30.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T15:15:30.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:30.371 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T15:15:30.386 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T15:15:30.386 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T15:15:30.396 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T15:15:30.396 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'}} 2024-09-21T15:15:30.396 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-21T15:15:30.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T15:15:30.739 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T15:15:30.739 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T15:15:30.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:15:31.037 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:15:31.055 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T15:15:29.917954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":187944,"kb_used_data":1224,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566043096,"statfs":{"total":579820584960,"available":579628130304,"internally_reserved":0,"allocated":1253376,"data_stored":1553680,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.104985"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.830605+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:15:24.830605+0000","last_peered":"2024-09-21T15:15:24.830605+0000","last_clean":"2024-09-21T15:15:24.830605+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:15:24.830605+0000","last_undegraded":"2024-09-21T15:15:24.830605+0000","last_fullsized":"2024-09-21T15:15:24.830605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.826574+0000","last_change":"2024-09-21T15:15:24.826574+0000","last_active":"2024-09-21T15:15:24.826574+0000","last_peered":"2024-09-21T15:15:24.826574+0000","last_clean":"2024-09-21T15:15:24.826574+0000","last_became_active":"2024-09-21T15:15:22.824152+0000","last_became_peered":"2024-09-21T15:15:22.824152+0000","last_unstale":"2024-09-21T15:15:24.826574+0000","last_undegraded":"2024-09-21T15:15:24.826574+0000","last_fullsized":"2024-09-21T15:15:24.826574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:54.629041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066240600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829674+0000","last_change":"2024-09-21T15:15:24.829674+0000","last_active":"2024-09-21T15:15:24.829674+0000","last_peered":"2024-09-21T15:15:24.829674+0000","last_clean":"2024-09-21T15:15:24.829674+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:15:24.829674+0000","last_undegraded":"2024-09-21T15:15:24.829674+0000","last_fullsized":"2024-09-21T15:15:24.829674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:48.076212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.427839+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:15:25.427839+0000","last_peered":"2024-09-21T15:15:25.427839+0000","last_clean":"2024-09-21T15:15:25.427839+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:15:25.427839+0000","last_undegraded":"2024-09-21T15:15:25.427839+0000","last_fullsized":"2024-09-21T15:15:25.427839+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.834145+0000","last_change":"2024-09-21T15:15:24.830325+0000","last_active":"2024-09-21T15:15:24.834145+0000","last_peered":"2024-09-21T15:15:24.834145+0000","last_clean":"2024-09-21T15:15:24.834145+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:15:24.834145+0000","last_undegraded":"2024-09-21T15:15:24.834145+0000","last_fullsized":"2024-09-21T15:15:24.834145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:55:21.434241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.511790+0000","last_change":"2024-09-21T15:15:25.511790+0000","last_active":"2024-09-21T15:15:25.511790+0000","last_peered":"2024-09-21T15:15:25.511790+0000","last_clean":"2024-09-21T15:15:25.511790+0000","last_became_active":"2024-09-21T15:15:22.828112+0000","last_became_peered":"2024-09-21T15:15:22.828112+0000","last_unstale":"2024-09-21T15:15:25.511790+0000","last_undegraded":"2024-09-21T15:15:25.511790+0000","last_fullsized":"2024-09-21T15:15:25.511790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:10:48.407777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069731800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.659947+0000","last_change":"2024-09-21T15:15:25.660255+0000","last_active":"2024-09-21T15:15:25.659947+0000","last_peered":"2024-09-21T15:15:25.659947+0000","last_clean":"2024-09-21T15:15:25.659947+0000","last_became_active":"2024-09-21T15:15:22.821654+0000","last_became_peered":"2024-09-21T15:15:22.821654+0000","last_unstale":"2024-09-21T15:15:25.659947+0000","last_undegraded":"2024-09-21T15:15:25.659947+0000","last_fullsized":"2024-09-21T15:15:25.659947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:55.119052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000470638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829906+0000","last_change":"2024-09-21T15:15:24.829906+0000","last_active":"2024-09-21T15:15:24.829906+0000","last_peered":"2024-09-21T15:15:24.829906+0000","last_clean":"2024-09-21T15:15:24.829906+0000","last_became_active":"2024-09-21T15:15:22.819834+0000","last_became_peered":"2024-09-21T15:15:22.819834+0000","last_unstale":"2024-09-21T15:15:24.829906+0000","last_undegraded":"2024-09-21T15:15:24.829906+0000","last_fullsized":"2024-09-21T15:15:24.829906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:32.673395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:15:23.818914+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:15:23.818914+0000","last_peered":"2024-09-21T15:15:23.818914+0000","last_clean":"2024-09-21T15:15:23.818914+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:15:23.818914+0000","last_undegraded":"2024-09-21T15:15:23.818914+0000","last_fullsized":"2024-09-21T15:15:23.818914+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31312,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340528,"statfs":{"total":96636764160,"available":96604700672,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:15:31.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:15:31.359 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:31.362 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:15:31.374 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T15:15:29.917954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":187944,"kb_used_data":1224,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566043096,"statfs":{"total":579820584960,"available":579628130304,"internally_reserved":0,"allocated":1253376,"data_stored":1553680,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.104985"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.830605+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:15:24.830605+0000","last_peered":"2024-09-21T15:15:24.830605+0000","last_clean":"2024-09-21T15:15:24.830605+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:15:24.830605+0000","last_undegraded":"2024-09-21T15:15:24.830605+0000","last_fullsized":"2024-09-21T15:15:24.830605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.826574+0000","last_change":"2024-09-21T15:15:24.826574+0000","last_active":"2024-09-21T15:15:24.826574+0000","last_peered":"2024-09-21T15:15:24.826574+0000","last_clean":"2024-09-21T15:15:24.826574+0000","last_became_active":"2024-09-21T15:15:22.824152+0000","last_became_peered":"2024-09-21T15:15:22.824152+0000","last_unstale":"2024-09-21T15:15:24.826574+0000","last_undegraded":"2024-09-21T15:15:24.826574+0000","last_fullsized":"2024-09-21T15:15:24.826574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:54.629041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066240600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829674+0000","last_change":"2024-09-21T15:15:24.829674+0000","last_active":"2024-09-21T15:15:24.829674+0000","last_peered":"2024-09-21T15:15:24.829674+0000","last_clean":"2024-09-21T15:15:24.829674+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:15:24.829674+0000","last_undegraded":"2024-09-21T15:15:24.829674+0000","last_fullsized":"2024-09-21T15:15:24.829674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:48.076212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.427839+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:15:25.427839+0000","last_peered":"2024-09-21T15:15:25.427839+0000","last_clean":"2024-09-21T15:15:25.427839+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:15:25.427839+0000","last_undegraded":"2024-09-21T15:15:25.427839+0000","last_fullsized":"2024-09-21T15:15:25.427839+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.834145+0000","last_change":"2024-09-21T15:15:24.830325+0000","last_active":"2024-09-21T15:15:24.834145+0000","last_peered":"2024-09-21T15:15:24.834145+0000","last_clean":"2024-09-21T15:15:24.834145+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:15:24.834145+0000","last_undegraded":"2024-09-21T15:15:24.834145+0000","last_fullsized":"2024-09-21T15:15:24.834145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:55:21.434241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.511790+0000","last_change":"2024-09-21T15:15:25.511790+0000","last_active":"2024-09-21T15:15:25.511790+0000","last_peered":"2024-09-21T15:15:25.511790+0000","last_clean":"2024-09-21T15:15:25.511790+0000","last_became_active":"2024-09-21T15:15:22.828112+0000","last_became_peered":"2024-09-21T15:15:22.828112+0000","last_unstale":"2024-09-21T15:15:25.511790+0000","last_undegraded":"2024-09-21T15:15:25.511790+0000","last_fullsized":"2024-09-21T15:15:25.511790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:10:48.407777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069731800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:25.659947+0000","last_change":"2024-09-21T15:15:25.660255+0000","last_active":"2024-09-21T15:15:25.659947+0000","last_peered":"2024-09-21T15:15:25.659947+0000","last_clean":"2024-09-21T15:15:25.659947+0000","last_became_active":"2024-09-21T15:15:22.821654+0000","last_became_peered":"2024-09-21T15:15:22.821654+0000","last_unstale":"2024-09-21T15:15:25.659947+0000","last_undegraded":"2024-09-21T15:15:25.659947+0000","last_fullsized":"2024-09-21T15:15:25.659947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:55.119052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000470638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:15:24.829906+0000","last_change":"2024-09-21T15:15:24.829906+0000","last_active":"2024-09-21T15:15:24.829906+0000","last_peered":"2024-09-21T15:15:24.829906+0000","last_clean":"2024-09-21T15:15:24.829906+0000","last_became_active":"2024-09-21T15:15:22.819834+0000","last_became_peered":"2024-09-21T15:15:22.819834+0000","last_unstale":"2024-09-21T15:15:24.829906+0000","last_undegraded":"2024-09-21T15:15:24.829906+0000","last_fullsized":"2024-09-21T15:15:24.829906+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:32.673395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:15:23.818914+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:15:23.818914+0000","last_peered":"2024-09-21T15:15:23.818914+0000","last_clean":"2024-09-21T15:15:23.818914+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:15:23.818914+0000","last_undegraded":"2024-09-21T15:15:23.818914+0000","last_fullsized":"2024-09-21T15:15:23.818914+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31356,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340484,"statfs":{"total":96636764160,"available":96604655616,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":245760,"data_stored":652519,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31312,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340528,"statfs":{"total":96636764160,"available":96604700672,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":192512,"data_stored":62170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":188416,"data_stored":62151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:15:31.375 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T15:15:31.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 21 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:fsid fa0b2d62-c633-4668-b5db-f1f6328d1af9 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-09-21T15:15:04.255598+0000 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-09-21T15:15:29.183600+0000 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:crush_version 5 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:full_ratio 0.95 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:backfillfull_ratio 0.9 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:nearfull_ratio 0.85 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:require_min_compat_client luminous 2024-09-21T15:15:31.674 INFO:teuthology.orchestra.run.smithi067.stdout:min_compat_client jewel 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:require_osd_release reef 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:stretch_mode_enabled false 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.06 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 20 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:max_osd 6 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6816/3600997419,v1:172.21.15.67:6817/3600997419] [v2:172.21.15.67:6818/3600997419,v1:172.21.15.67:6819/3600997419] exists,up 6194913c-995a-4f8b-b2e4-1ae24c7fc1c6 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6808/1817390160,v1:172.21.15.67:6809/1817390160] [v2:172.21.15.67:6810/1817390160,v1:172.21.15.67:6811/1817390160] exists,up 496a596a-f018-4010-bd93-6df3cfbef67f 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/3371454484,v1:172.21.15.67:6801/3371454484] [v2:172.21.15.67:6802/3371454484,v1:172.21.15.67:6803/3371454484] exists,up 0131c1a7-80b4-448b-b7b2-89e6ba9ec584 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6808/1069930087,v1:172.21.15.142:6809/1069930087] [v2:172.21.15.142:6810/1069930087,v1:172.21.15.142:6811/1069930087] exists,up edacb78d-2a42-4dd8-a648-86031fd9ad18 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6816/3606282773,v1:172.21.15.142:6817/3606282773] [v2:172.21.15.142:6818/3606282773,v1:172.21.15.142:6819/3606282773] exists,up bd5f5e45-2c13-432b-9aba-69c731ca2248 2024-09-21T15:15:31.675 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6800/3327577487,v1:172.21.15.142:6801/3327577487] [v2:172.21.15.142:6802/3327577487,v1:172.21.15.142:6803/3327577487] exists,up 52ceefef-20a8-41fb-ab77-053021d17105 2024-09-21T15:15:31.688 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6816/3600997419,v1:172.21.15.67:6817/3600997419] [v2:172.21.15.67:6818/3600997419,v1:172.21.15.67:6819/3600997419] exists,up 6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'osd.1 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6808/1817390160,v1:172.21.15.67:6809/1817390160] [v2:172.21.15.67:6810/1817390160,v1:172.21.15.67:6811/1817390160] exists,up 496a596a-f018-4010-bd93-6df3cfbef67f', 'osd.2 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/3371454484,v1:172.21.15.67:6801/3371454484] [v2:172.21.15.67:6802/3371454484,v1:172.21.15.67:6803/3371454484] exists,up 0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'osd.3 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6808/1069930087,v1:172.21.15.142:6809/1069930087] [v2:172.21.15.142:6810/1069930087,v1:172.21.15.142:6811/1069930087] exists,up edacb78d-2a42-4dd8-a648-86031fd9ad18', 'osd.4 up in weight 1 up_from 12 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6816/3606282773,v1:172.21.15.142:6817/3606282773] [v2:172.21.15.142:6818/3606282773,v1:172.21.15.142:6819/3606282773] exists,up bd5f5e45-2c13-432b-9aba-69c731ca2248', 'osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6800/3327577487,v1:172.21.15.142:6801/3327577487] [v2:172.21.15.142:6802/3327577487,v1:172.21.15.142:6803/3327577487] exists,up 52ceefef-20a8-41fb-ab77-053021d17105'] 2024-09-21T15:15:31.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-21T15:15:31.795 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:31.796 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mon.a", 2024-09-21T15:15:31.796 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster": "ceph", 2024-09-21T15:15:31.796 INFO:teuthology.orchestra.run.smithi067.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T15:15:31.796 INFO:teuthology.orchestra.run.smithi067.stdout: "admin_socket_mode": "", 2024-09-21T15:15:31.796 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-21T15:15:31.796 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_client_required": "cephx, none", 2024-09-21T15:15:31.796 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_cluster_required": "cephx", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_debug": "true", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_service_required": "cephx", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_supported": "cephx", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio": "true", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_reap_max": "16", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_async_discard": "false", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_block_size": "4096", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio": "false", 2024-09-21T15:15:31.797 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_enable_discard": "false", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_flock_retry": "3", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_inject_crash": "0", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring": "false", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring_hipri": "false", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T15:15:31.798 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_type": "", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_allocator": "hybrid", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_buffered_io": "true", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T15:15:31.799 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_replay_recovery": "false", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_sync_write": "false", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_allocator": "hybrid", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T15:15:31.800 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_create": "true", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_create": "false", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_path": "", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_size": "0", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_path": "", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_size": "107374182400", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_create": "false", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_path": "", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T15:15:31.801 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs": "true", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_autotune": "true", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size": "0", 2024-09-21T15:15:31.802 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_type": "2q", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_clone_cow": "true", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T15:15:31.803 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_mode": "none", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_freelist": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T15:15:31.804 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T15:15:31.805 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fail_eio": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T15:15:31.806 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T15:15:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size": "0", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T15:15:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_mem": "512", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T15:15:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_tracing": "false", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T15:15:31.810 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_cluster_require_version": "2", 2024-09-21T15:15:31.811 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_require_signatures": "false", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_require_version": "2", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_service_require_signatures": "false", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_service_require_version": "2", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_sign_messages": "true", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "chdir": "", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_acl_type": "", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_asio_thread_count": "2", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_cache_mid": "0.750000", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_cache_size": "16384", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_caps_release_delay": "5", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_check_pool_perm": "true", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_force_sync_read": "false", 2024-09-21T15:15:31.812 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_getattr_caps": "false", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_inject_features": "", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_die_on_failed_remount": "false", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_dirsize_rbytes": "true", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_force_lazyio": "false", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_fs": "", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_inject_release_failure": "false", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_max_inline_size": "4096", 2024-09-21T15:15:31.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mds_namespace": "", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_metadata": "", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_gid": "-1", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_timeout": "300", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_uid": "-1", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mountpoint": "/", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_notify_timeout": "10", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc": "true", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_objects": "1000", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_size": "209715200", 2024-09-21T15:15:31.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_permissions": "true", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_quota": "true", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_quota_df": "true", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_max_bytes": "0", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_max_periods": "4", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_min": "131072", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_reconnect_stale": "false", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_shutdown_timeout": "30", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_snapdir": ".snap", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_tick_interval": "1", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_trace": "", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T15:15:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_use_faked_inos": "false", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "client_use_random_mds": "false", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog": "false", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog_port": "12201", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_monitors": "default=true", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog": "false", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog_level": "info", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "-", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_network": "", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_network_interface": "", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_isal": "false", 2024-09-21T15:15:31.816 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_level": "5", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zstd_level": "1", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_seastar_smp": "1", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location": "", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location_hook": "", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location_hook_timeout": "10", 2024-09-21T15:15:31.817 INFO:teuthology.orchestra.run.smithi067.stdout: "daemonize": "false", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "daos_pool": "tank", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_alienstore": "0/5", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asok": "1/5", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asok_assert_abort": "false", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_auth": "20/20", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bdev": "1/3", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bluefs": "1/5", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bluestore": "1/5", 2024-09-21T15:15:31.818 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_buffer": "0/1", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cephsqlite": "0/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_client": "0/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_compressor": "1/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_context": "0/1", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crush": "1/1", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crypto": "1/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cyanstore": "0/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_dpdk": "1/5", 2024-09-21T15:15:31.819 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_eventtrace": "1/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_filer": "0/1", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_filestore": "1/3", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_finisher": "1/1", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_fuse": "1/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_javaclient": "1/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_journal": "1/3", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_journaler": "0/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_kstore": "1/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_leveldb": "4/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_lockdep": "0/1", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mclock": "1/5", 2024-09-21T15:15:31.820 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_balancer": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_locker": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_log": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_migrator": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_memstore": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mgr": "2/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mgrc": "1/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mon": "20/20", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_monc": "0/10", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_ms": "1/1", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_none": "0/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objclass": "0/5", 2024-09-21T15:15:31.821 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objectcacher": "0/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objecter": "0/1", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_optracker": "0/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_osd": "1/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_paxos": "20/20", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_perfcounter": "1/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_prioritycache": "1/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rados": "0/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd": "0/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_replay": "0/5", 2024-09-21T15:15:31.822 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_refs": "0/0", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_reserver": "1/1", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw": "1/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_access": "1/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_flight": "1/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_sync": "1/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rocksdb": "4/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore": "0/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_backref": "0/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_cache": "0/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_device": "0/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_epm": "0/5", 2024-09-21T15:15:31.823 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_journal": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_lba": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_odata": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_omap": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_onode": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_t": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_tm": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_striper": "0/1", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_test": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_throttle": "1/1", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_timer": "0/1", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_tp": "0/5", 2024-09-21T15:15:31.824 INFO:teuthology.orchestra.run.smithi067.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "device_failure_prediction_mode": "none", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_graylog": "false", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_journald": "false", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_stderr": "true", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_syslog": "false", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event_tracing": "false", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_http_port": "9926", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_prio_limit": "5", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T15:15:31.825 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_sort_metrics": "true", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_stats_period": "5", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "fake_statfs_for_testing": "0", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "fatal_signal_handlers": "true", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filer_max_purge_ops": "10", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filer_max_truncate_ops": "128", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_blackhole": "false", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_btrfs_snap": "true", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_caller_concurrency": "10", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T15:15:31.826 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_omap_check": "false", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_verify_split": "false", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_dump_file": "", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fadvise": "true", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fail_eio": "true", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fd_cache_size": "128", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fiemap": "false", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T15:15:31.827 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_inject_stall": "0", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_parallel": "false", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_trailing": "false", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_writeahead": "false", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_kill_at": "0", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T15:15:31.828 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_merge_threshold": "-10", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_odsync_write": "false", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_backend_path": "", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T15:15:31.829 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_threads": "2", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_punch_hole": "false", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_ops": "50", 2024-09-21T15:15:31.830 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_seek_data_hole": "false", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_sloppy_crc": "false", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_splice": "false", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_split_multiple": "2", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_split_rand_factor": "20", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_update_to": "1000", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T15:15:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_xfs_extsize": "false", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_zfs_snap": "false", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "fio_dir": "/tmp/fio", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_allow_other": "true", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_big_writes": "true", 2024-09-21T15:15:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_debug": "false", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_default_permissions": "false", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_disable_pagecache": "false", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_max_write": "0", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_multithreaded": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_require_active_mds": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_set_user_groups": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_move": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_read": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_write": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "gss_target_name": "ceph", 2024-09-21T15:15:31.833 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_file": "", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_inject_failure": "0", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_interval": "5", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "host": "smithi067", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T15:15:31.834 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "inject_early_sigterm": "false", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "jaeger_agent_port": "6799", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "jaeger_tracing_enable": "false", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_aio": "true", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_align_min_size": "65536", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_block_align": "true", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_block_size": "4096", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_dio": "true", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_discard": "false", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_force_aio": "false", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_ignore_corruption": "false", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T15:15:31.835 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_max_write_entries": "100", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_replay_from": "0", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_write_header_frequency": "0", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_zero_on_create": "false", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_prefetch_periods": "10", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_prezero_periods": "5", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_write_head_interval": "15", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "keyfile": "", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_backend": "rocksdb", 2024-09-21T15:15:31.836 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_max_bytes": "67108864", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_max_ops": "512", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_onode_map_size": "1024", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_sync_transaction": "false", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_block_size": "65536", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_bloom_size": "0", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_cache_size": "536870912", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_compact_on_mount": "false", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_compression": "false", 2024-09-21T15:15:31.837 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_log": "", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_max_open_files": "0", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_paranoid": "false", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "librados_thread_count": "2", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "lockdep": "false", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "lockdep_force_backtrace": "false", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "log_coarse_timestamps": "true", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "log_flush_on_exit": "false", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "log_graylog_port": "12201", 2024-09-21T15:15:31.838 INFO:teuthology.orchestra.run.smithi067.stdout: "log_max_new": "1000", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_max_recent": "10000", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_stderr_prefix": "", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": "true", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_graylog": "false", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_journald": "false", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_stderr": "false", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_syslog": "false", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_action_on_write_error": "1", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_alternate_name_max": "8192", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_asio_thread_count": "2", 2024-09-21T15:15:31.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_export_pin": "true", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_interval": "10", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_max": "-1", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_max_until": "-1", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_merge_size": "50", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_mode": "0", 2024-09-21T15:15:31.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_bits": "3", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_size": "10000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_mid": "0.700000", 2024-09-21T15:15:31.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_interval": "1", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_auth_pins": "false", 2024-09-21T15:15:31.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_frag": "false", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_scatterstat": "false", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_subtrees": "false", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_default_dir_hash": "2", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_defer_session_stale": "true", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_max_entries": "0", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_prefetch": "true", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T15:15:31.843 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_early_reply": "true", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_enable_op_tracker": "true", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_enforce_unique_name": "true", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T15:15:31.844 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_health_dummy": "false", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_join_fs": "", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_journal_format": "1", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_create_at": "0", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_export_at": "0", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_import_at": "0", 2024-09-21T15:15:31.845 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_at": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_link_at": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_openc_at": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_rename_at": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_max_events": "-1", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_max_segments": "128", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_pause": "false", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_segment_size": "0", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T15:15:31.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_completed_requests": "100000", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_export_size": "20971520", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_file_recover": "32", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_files": "64", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_ops": "8192", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_metrics_update_interval": "2", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_min_caps_per_client": "100", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_numa_node": "-1", 2024-09-21T15:15:31.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_duration": "600", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_size": "20", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_log_threshold": "5", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_ping_grace": "15", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_ping_interval": "5", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_caps": "30000", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T15:15:31.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_replay_interval": "1.000000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_root_ino_gid": "0", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_root_ino_uid": "0", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T15:15:31.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_shutdown_check": "0", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_skip_ino": "0", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_min_uid": "0", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_rstat": "false", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_symlink_recovery": "true", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_thrash_exports": "0", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_thrash_fragments": "0", 2024-09-21T15:15:31.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_tick_interval": "5.000000", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_valgrind_exit": "false", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_verify_backtrace": "1", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_verify_scatter": "false", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_wipe_sessions": "false", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool_debug": "false", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_page_set": "false", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_page_size": "65536", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_bytes": "134217728", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_messages": "512", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T15:15:31.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_disabled_modules": "", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mds_messages": "128", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mon_messages": "128", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_osd_messages": "8192", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_pool": "true", 2024-09-21T15:15:31.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_standby_modules": "true", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_stats_period": "5", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_stats_threshold": "5", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_tick_period": "2", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_allow_pool_delete": "true", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_allow_pool_size_one": "true", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_bytes": "104857600", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_directed_command_retry": "5", 2024-09-21T15:15:31.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_file_level": "debug", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T15:15:31.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_journald": "false", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_start": "false", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_trim": "true", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T15:15:31.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cpu_threads": "4", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_avail_crit": "5", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_avail_warn": "5", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_json": "false", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_extra_checks": "false", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T15:15:31.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_quincy": "false", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_down_mkfs_grace": "120", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_election_default_strategy": "1", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_election_timeout": "5.000000", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_enable_op_tracker": "true", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_fake_pool_delete": "false", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_force_quorum_join": "false", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_detail_to_clog": "false", 2024-09-21T15:15:31.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_log_update_period": "5", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_max_detail": "50", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog": "true", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_host": "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.142", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_host_override": "", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_initial_members": "", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease": "5.000000", 2024-09-21T15:15:31.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_full_interval": "50", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_max": "10000", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_max_summary": "50", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_log_epochs": "500", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_osd": "10000", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T15:15:31.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_memory_autotune": "true", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_memory_target": "2147483648", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_digest_period": "5", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_complaint_time": "30", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_duration": "600", 2024-09-21T15:15:31.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_size": "20", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_log_threshold": "5", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_cache_size": "500", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T15:15:31.861 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T15:15:31.862 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_report_timeout": "900", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T15:15:31.863 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_max_osds": "4", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T15:15:31.864 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_interval": "86400", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_max_keys": "100", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_timeout": "300", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_session_timeout": "300", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_smart_report_timeout": "5", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_pool_size": "4", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_debug": "false", 2024-09-21T15:15:31.865 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_tick_interval": "5", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T15:15:31.866 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_older_version": "true", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T15:15:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "monmap": "", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_tracing_enabled": "false", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_op_threads": "3", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_cm": "false", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_device_name": "", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T15:15:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_sl": "3", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_type": "ib", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_reap_threshold": "5", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_before_connect": "false", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_ipv4": "true", 2024-09-21T15:15:31.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_ipv6": "false", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_msgr1": "true", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_msgr2": "true", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_port_max": "7568", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_port_min": "6800", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_retry_count": "3", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_retry_delay": "5", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_client": "false", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mds": "false", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mgr": "false", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mon": "false", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_osd": "false", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_client_mode": "crc secure", 2024-09-21T15:15:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_cluster_type": "", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_compress_secure": "false", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_crc_data": "true", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_crc_header": "true", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_bug": "true", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_old_message": "true", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T15:15:31.871 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T15:15:31.872 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T15:15:31.872 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T15:15:31.872 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T15:15:31.873 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T15:15:31.873 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T15:15:31.873 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hugepages": "", 2024-09-21T15:15:31.873 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T15:15:31.873 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T15:15:31.873 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_lro": "true", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_pmd": "", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_port_id": "0", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dump_on_send": "false", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_type": "", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_network_congestion": "0", 2024-09-21T15:15:31.874 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_socket_failures": "5000", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_max_accept_failures": "4", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_max_backoff": "15.000000", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compress_mode": "none", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_pq_min_cost": "65536", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_public_type": "", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_service_mode": "crc secure", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T15:15:31.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_nodelay": "true", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_type": "async+posix", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "no_config_file": "false", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inflight_ops": "1024", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_timeout": "10.000000", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "objectstore_blackhole": "false", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "openssl_engine_opts": "", 2024-09-21T15:15:31.876 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_max_ops": "4", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_slop": "0.020000", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_asio_thread_count": "2", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_scan_max": "512", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_scan_min": "64", 2024-09-21T15:15:31.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_peering": "false", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_beacon_report_interval": "300", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_duration": "30", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_blkin_trace_all": "false", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T15:15:31.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_update_on_start": "true", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_message_cap": "256", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_op_priority": "63", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_watch_timeout": "30", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_command_max_records": "256", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_compact_on_start": "false", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T15:15:31.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_update_on_start": "true", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T15:15:31.880 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_op_order": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_shutdown": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T15:15:31.881 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_default_notify_timeout": "30", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_enable_op_tracker": "true", 2024-09-21T15:15:31.882 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_info": "true", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown": "true", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_function_tracing": "false", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_grace": "20", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_interval": "6", 2024-09-21T15:15:31.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_stale": "600", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal_size": "5120", 2024-09-21T15:15:31.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_kill_backfill_at": "0", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_cache_size": "50", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_dedup": "true", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_message_max": "40", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_attr_name_len": "100", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_attr_size": "0", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_backfills": "1", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_markdown_count": "5", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_markdown_period": "600", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_name_len": "2048", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T15:15:31.885 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_size": "134217728", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pgls": "1024", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_push_cost": "8388608", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_push_objects": "10", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_scrubs": "3", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_write_size": "90", 2024-09-21T15:15:31.886 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_profile": "balanced", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T15:15:31.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_base": "805306368", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target": "4294967296", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target_autotune": "false", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_min_recovery_priority": "0", 2024-09-21T15:15:31.888 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_report_interval": "5", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_num_cache_shards": "32", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_node": "-1", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objecter_finishers": "1", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore": "bluestore", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore_fuse": "false", 2024-09-21T15:15:31.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore_tracing": "false", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_duration": "600", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_size": "20", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_log_threshold": "5", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards": "0", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T15:15:31.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_thread_timeout": "15", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_open_classes_on_start": "true", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_os_flags": "0", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_peering_op_priority": "255", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_delete_priority": "5", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T15:15:31.891 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_crimson": "false", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T15:15:31.892 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flags": "0", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_min_size": "0", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_size": "2", 2024-09-21T15:15:31.893 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_type": "replicated", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_cost": "20971520", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active": "0", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T15:15:31.894 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_op_priority": "3", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_priority": "5", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_repair_during_recovery": "false", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_requested_scrub_priority": "120", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T15:15:31.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_chunk_max": "25", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_cost": "52428800", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_end_hour": "0", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T15:15:31.896 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_priority": "5", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_reservation_timeout": "5000", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_skip_data_digest": "false", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_smart_report_timeout": "5", 2024-09-21T15:15:31.897 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_priority": "5", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_target_transaction_size": "30", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T15:15:31.898 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tracing": "false", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_use_stale_snap": "false", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_kill_at": "0", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_max_join_drift": "10", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_min": "500", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_min_wait": "0.050000", 2024-09-21T15:15:31.899 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_max": "500", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_min": "250", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_stash_full_interval": "25", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_trim_max": "500", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_trim_min": "250", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "perf": "true", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "-", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrv": "", 2024-09-21T15:15:31.900 INFO:teuthology.orchestra.run.smithi067.stdout: "public_bind_addr": "-", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "public_network": "", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "public_network_interface": "", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "qat_compressor_enabled": "false", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_mon_op_timeout": "0", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_osd_op_timeout": "0", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_tracing": "false", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_atime_update_interval": "60", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T15:15:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache": "true", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_policy": "writearound", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_size": "33554432", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_compression_hint": "none", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T15:15:31.902 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_clone_format": "auto", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_data_pool": "", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_features": "61", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_format": "2", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_map_options": "", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_order": "22", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_pool": "rbd", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_stripe_count": "0", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T15:15:31.903 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_io_scheduler": "simple", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_order": "24", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_pool": "", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_splay_width": "4", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T15:15:31.904 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T15:15:31.905 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T15:15:31.906 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_op_threads": "1", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_plugins": "", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T15:15:31.907 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T15:15:31.908 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_tracing": "false", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_validate_names": "true", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_validate_pool": "true", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "restapi_base_url": "", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "restapi_log_level": "", 2024-09-21T15:15:31.909 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_admin_entry": "admin", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_backend_store": "rados", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_barbican_url": "", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_beast_enable_async": "true", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T15:15:31.910 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_enabled": "true", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_lru_size": "10000", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_config_store": "rados", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_content_length_compat": "false", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cross_domain_policy": "", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T15:15:31.911 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T15:15:31.912 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T15:15:31.913 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T15:15:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_window": "30", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T15:15:31.915 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dns_name": "", 2024-09-21T15:15:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dns_s3website_name": "", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_ops_log": "false", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_static_website": "false", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_usage_log": "false", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_expose_bucket": "false", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_extended_http_attrs": "", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_filter": "none", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T15:15:31.917 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_objs": "32", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T15:15:31.918 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_init_timeout": "300", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_password": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_project": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_token": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_user": "", 2024-09-21T15:15:31.919 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_api_version": "2", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_url": "", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T15:15:31.920 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_objs": "32", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_worker": "3", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_uri": "ldaps://", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T15:15:31.921 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_http_headers": "", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attr_size": "0", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T15:15:31.922 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_listing_results": "1000", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_notify_retries": "10", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T15:15:31.923 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T15:15:31.924 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_num_control_oids": "8", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_numa_node": "-1", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_tracing": "false", 2024-09-21T15:15:31.925 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_token": "", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_url": "", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_rados": "false", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T15:15:31.926 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_print_continue": "true", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_tracing": "false", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm": "", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm_id": "", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_region": "", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T15:15:31.927 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_request_uri": "", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_resolve_cname": "false", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_run_sync_thread": "true", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T15:15:31.928 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_script_uri": "", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_service_provider_name": "", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_client_id": "", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_client_secret": "", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_entry": "sts", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_key": "sts", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T15:15:31.929 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_auth_url": "", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_custom_header": "", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_need_stats": "true", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_tenant_name": "", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_url": "", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_lease_period": "120", 2024-09-21T15:15:31.930 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_thread_pool_size": "512", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_comment": "", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_createby": "", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_encoding": "", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_flag": "false", 2024-09-21T15:15:31.931 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_origin": "", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_tracker": "", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_max_shards": "32", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_use_opa_authz": "false", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T15:15:31.932 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_unique_email": "true", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_verify_ssl": "true", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone": "", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone_id": "", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup": "", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup_id": "", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_block_size": "4096", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T15:15:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_partition_filters": "false", 2024-09-21T15:15:31.934 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_perf": "false", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "run_dir": "/var/run/ceph", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_block_create": "true", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_device_size": "53687091200", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T15:15:31.935 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_main_device_type": "SSD", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_segment_size": "67108864", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "set_keepcaps": "false", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "setgroup": "", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "setuser": "", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "setuser_match_path": "", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "thp": "false", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "threadpool_default_timeout": "60", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout: "throttler_perf_counter": "true" 2024-09-21T15:15:31.936 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:31.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T15:15:32.084 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a: {} 2024-09-21T15:15:32.099 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b: {} 2024-09-21T15:15:32.116 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c: {} 2024-09-21T15:15:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:15:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:15:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:15:32.131 DEBUG:teuthology.orchestra.run.smithi142:> type ceph-objectstore-tool 2024-09-21T15:15:32.138 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T15:15:32.138 DEBUG:teuthology.orchestra.run.smithi067:> type ceph-objectstore-tool 2024-09-21T15:15:32.144 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T15:15:32.144 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T15:15:32.162 INFO:tasks.workunit:Pulling workunits from ref a477b1894d5c5a613f23260ca0e6b96d79652ba7 2024-09-21T15:15:32.163 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T15:15:32.163 DEBUG:teuthology.orchestra.run.smithi146:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T15:15:32.163 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T15:15:32.163 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:15:32.163 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:15:32.163 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T15:15:32.164 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.708862 2024-09-21T15:15:32.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.7088622312319217 2024-09-21T15:15:32.164 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T15:15:32.164 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T15:15:32.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:15:32.165 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T15:15:32.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:32.165 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T15:15:32.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:15:32.167 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:32.162+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:15:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T15:15:32.173 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T15:15:32.220 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T15:15:32.220 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T15:15:32.227 INFO:tasks.workunit:timeout=3h 2024-09-21T15:15:32.227 INFO:tasks.workunit:cleanup=True 2024-09-21T15:15:32.228 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a477b1894d5c5a613f23260ca0e6b96d79652ba7 2024-09-21T15:15:32.236 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T15:15:32.266 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:32.266+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:32.283 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:32.283 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:32.284 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:32.284 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:32.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:32.308 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:32.343 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:32.367 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:32.366+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:32.379 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:32.409 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:32.409 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:32.414 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:32.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:32.440 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:32.462 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:32.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:32.468 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:32.465+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:32.522 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:32.522 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:32.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:32.568 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:32.566+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:32.631 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:32.632 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:32.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:32.682 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:32.682+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:32.769 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:32.764+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:32.799 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:32.799 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:32.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:32.871 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:32.868+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:32.912 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:32.913 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:32.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:32.971 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:32.970+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:33.032 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.032 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:33.086 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:33.086+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:33.173 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:33.168+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:33.197 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.197 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:15:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 0.708862 (8464552) 2024-09-21T15:15:33.274 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:33.274+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:33.311 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.311 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:33.375 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:33.372+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:33.439 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.440 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.449 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:33.477 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:33.472+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:33.555 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.555 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:33.607 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:33.604+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:33.678 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:33.678+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:33.715 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.715 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.724 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:33.779 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:33.776+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:33.837 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.837 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.846 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:33.880 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:33.878+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:33.955 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:33.956 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:33.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:33.981 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:33.978+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:34.076 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:34.076 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:34.082 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:34.078+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:34.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:34.182 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:34.182+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:34.195 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:34.195 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:34.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:34.284 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:34.282+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:34.309 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:34.310 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:34.320 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:34.384 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:34.380+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:34.425 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:34.425 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:34.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:34.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:15:34.485 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:34.480+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:34.552 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:34.552 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:34.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:34.586 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:34.586+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:34.614 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:34.637 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:34.656 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:34.671 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:34.671 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:34.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:34.684 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:34.687 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:34.684+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:34.707 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:34.729 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:34.731 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:34.731 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:34.731 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:34.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:34.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:34.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:34.785 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.958238024, 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:34.786 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:34.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.957623163000001, 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:34.790 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:34.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:34.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.965768008, 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:34.794 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:34.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:34.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:15:34.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:34.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:34.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.960879694999999, 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:34.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:15:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.9556130760000006, 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:34.804 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:15:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:15:34.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:15:34.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:15:34.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.9533390619999995, 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:34.809 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:34.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:34.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:34.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:34.813 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:34.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:34.815 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:34.814+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:34.916 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:34.912+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:34.968 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:34.969 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:34.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:35.017 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:35.012+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:35.087 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:35.087 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:35.087 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:35.087 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:35.087 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:35.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:35.117 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:35.114+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:35.211 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:35.211 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.427647055000001, 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:15:35.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:15:35.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.218 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.293836684999999, 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:15:35.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:15:35.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.223 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.293777458000001, 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:15:35.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:15:35.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.179454629999999, 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:15:35.229 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:15:35.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:15:35.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971886769000001, 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.235 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.240 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971716829000002, 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:15:35.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971687929000002, 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.246 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:15:35.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:15:35.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:15:35.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971636453999999, 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:15:35.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971457764, 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:15:35.257 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:15:35.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:15:35.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971374955999998, 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:15:35.263 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:15:35.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.9712171, 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:15:35.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:15:35.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.274 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971092388999999, 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:15:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:15:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.280 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.971062336999999, 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:35.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:15:35.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:15:35.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:15:35.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.524854407999999, 2024-09-21T15:15:35.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:15:35.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.524813799, 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:15:35.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:15:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.524762386999999, 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:15:35.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:15:35.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.524577424, 2024-09-21T15:15:35.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:15:35.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.524547892000001, 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.524505196, 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:15:35.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:15:35.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.524263523999998, 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:15:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.324 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:35.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:15:35.326 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:35.320+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:35.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:35.427 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:35.426+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:35.484 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:35.485 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:35.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:35.528 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:35.524+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:35.592 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:35.592 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:35.592 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:35.592 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:35.592 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:35.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:35.628 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:35.626+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:35.707 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:35.707 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.887612687000001, 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:35.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:15:35.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:35.712 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:35.717 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:35.762 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:35.760+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:35.830 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:35.830+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:35.835 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:35.835 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:35.835 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:35.835 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:35.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:35.932 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:35.930+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:35.950 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:35.950 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:35.950 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:35.950 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:35.950 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:35.961 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:36.032 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:36.028+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:36.066 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:36.066 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:36.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:36.066 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:15:36.066 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:36.075 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:36.133 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:36.130+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:36.229 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:36.229 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:36.229 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:36.229 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:36.233 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:36.230+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:15:36.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:36.334 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:36.334+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:36.342 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:36.343 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:36.343 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:36.343 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:36.343 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:36.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:36.435 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:36.434+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 12.634445598999999, 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:15:36.455 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:15:36.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:15:36.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:15:36.459 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.624889015000001, 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:15:36.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:15:36.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:15:36.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:15:36.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.464 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:36.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:36.536 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:36.532+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:36.567 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:36.567 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:36.567 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:36.567 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:36.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:36.636 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:36.634+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:36.728 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:36.728 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:36.728 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:36.728 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:36.728 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:36.737 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:36.732+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:36.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:36.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:15:36.838 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:36.832+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:36.847 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:36.847 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:36.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.949216838000002, 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.851 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.927052660000001, 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.855 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.925606318, 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:15:36.859 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.924632762000002, 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:15:36.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:15:36.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:15:36.863 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.922205902000002, 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:15:36.867 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.921498087, 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:15:36.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.848980562000001, 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.871 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:36.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:36.875 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.848187855999999, 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:36.879 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.847021455, 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:15:36.883 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.845230594, 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.814614215999999, 2024-09-21T15:15:36.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:15:36.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:15:36.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:15:36.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.812646006000001, 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.891 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.810892986999999, 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:15:36.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.809508347000001, 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.899 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.608659893999999, 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:15:36.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.607894961, 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:15:36.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:15:36.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:15:36.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.606884571999998, 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:36.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:15:36.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:15:36.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.606024165000001, 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:36.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.604694294000002, 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.603234874999998, 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:36.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:15:36.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:36.926 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:36.927 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:36.927 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:36.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:36.938 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:36.936+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:36.966 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:36.988 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:37.039 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:37.036+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:37.039 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.039 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:37.140 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:37.136+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:37.152 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.153 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:37.240 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:37.236+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:37.341 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:37.336+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:37.386 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.387 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:37.442 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:37.440+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:37.505 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.505 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:37.543 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:37.542+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:37.622 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.622 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:37.644 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:37.642+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:37.740 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.740 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.745 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:37.740+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:37.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:37.846 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:37.840+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:37.907 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:37.908 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:37.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:37.946 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:37.946+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:38.025 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.025 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:38.078 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:38.078+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:38.149 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:38.146+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:38.153 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.153 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.163 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:38.232 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:15:38.233 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:15:38.233 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T15:15:38.233 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:15:38.233 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:15:38.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-21T15:15:38.250 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:38.244+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:38.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:15:38.271 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.271 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:38.335 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:38.335 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T15:15:38.335 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:38.350 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:38.350+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:38.386 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.386 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.396 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:38.451 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:38.448+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:38.506 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.506 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.516 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:38.552 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:38.550+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:38.621 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.635 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.636 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:38.653 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:38.648+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:38.744 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.744 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:38.754 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:38.750+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:38.854 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:38.854+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:38.908 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:38.908 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:38.918 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:38.954 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:38.954+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:39.026 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:39.026 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:39.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:15:39.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:39.074 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:39.074+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:39.155 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:39.155 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:39.156 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:39.154+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:39.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:39.186 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:39.207 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:39.227 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:15:39.251 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:39.257 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:39.254+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:39.274 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:39.276 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:39.276 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:39.276 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:39.276 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:39.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:39.297 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:39.299 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:39.299 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:39.299 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:39.299 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:39.299 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:39.299 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:39.358 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:39.352+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:39.397 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:39.398 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:39.398 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:39.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:39.459 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:39.458+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:39.519 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:39.519 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.691980609000002, 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:15:39.523 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.691365747999999, 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:15:39.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:15:39.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.699510592999999, 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:15:39.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:15:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:15:39.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.694622280000001, 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:15:39.533 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:15:39.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.689355661, 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:15:39.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.687081646999999, 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:39.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:15:39.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:15:39.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.546 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.547 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.547 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:39.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:39.559 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:39.556+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:39.657 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:39.657 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:39.657 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:39.660 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:39.656+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:39.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:39.762 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:39.756+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:39.777 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:39.778 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:39.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:39.862 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:39.862+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.116031113999998, 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:15:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.982220743999999, 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:15:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.982161517000002, 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:15:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:15:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:15:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:15:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.867838688999999, 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.660270828000002, 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:15:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:15:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:15:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.660100887999999, 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:15:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:15:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:15:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.660071987999999, 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.935 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.660020512999999, 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.936 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:15:39.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.941 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.659841823000001, 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:39.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:15:39.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:15:39.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.659759014999999, 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.947 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.659601159000001, 2024-09-21T15:15:39.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:15:39.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:15:39.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:15:39.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.659476447999999, 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:15:39.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:15:39.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.963 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.659446396, 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:15:39.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:15:39.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.213238467, 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.969 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:15:39.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:15:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:15:39.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.213197858000001, 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.975 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:39.980 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.213146446, 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:15:39.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.212961483000001, 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:15:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:15:39.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:15:39.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.212931951000002, 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:39.997 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.212889255, 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:39.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:15:39.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:15:40.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:15:40.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:15:40.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.212647582999999, 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:40.003 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:15:40.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:15:40.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:15:40.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:15:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:40.008 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:40.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:40.010 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:40.004+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:40.110 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:40.108+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:40.122 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:40.122 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:40.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:40.211 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:40.210+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:40.242 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:40.242 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:40.242 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:40.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:40.311 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:40.308+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:40.408 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.588667311999998, 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:40.413 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:40.414 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:40.414+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:40.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:40.515 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:40.512+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:40.534 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:40.534 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:40.534 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:40.534 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:40.544 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:40.615 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:40.614+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:40.646 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:40.646 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:40.646 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:40.646 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:40.646 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:40.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:40.717 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:40.718+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:40.763 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:40.763 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:40.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:40.763 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:15:40.763 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:40.773 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:40.817 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:40.812+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:40.878 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:40.878 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:40.878 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:40.879 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:40.888 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:40.917 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:40.918+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:41.019 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:41.018+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:41.041 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:41.041 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:41.041 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:41.041 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:41.041 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:41.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:41.120 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:41.116+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:41.160 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:41.160 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:41.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.339582636999999, 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:15:41.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 16.330026053000001, 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:15:41.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:15:41.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.169 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:41.171 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:41.220 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:41.216+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:41.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:15:41.279 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:41.279 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:41.279 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:41.279 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:41.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:41.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:15:41.321 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:41.318+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:41.422 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:41.422+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:41.443 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:41.443 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:41.443 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:41.443 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:41.443 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:41.453 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:41.461 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:41.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:41.497 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:41.523 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:41.520+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:41.526 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:41.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:41.563 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:41.563 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.665291836000002, 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:15:41.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.643127658000001, 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:15:41.568 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:15:41.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:15:41.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.641681316, 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:15:41.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.640707760000002, 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:15:41.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:15:41.581 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.638280900000002, 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:41.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:15:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.637573085, 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:15:41.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:15:41.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.565055560000001, 2024-09-21T15:15:41.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.564262853999999, 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:15:41.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.563096453, 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:41.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:15:41.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.561305592, 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.603 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:41.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:15:41.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:15:41.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.530689213999999, 2024-09-21T15:15:41.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:15:41.611 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.528721004000001, 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:15:41.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.526967984999999, 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:15:41.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:15:41.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.525583345000001, 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:15:41.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.324734891999999, 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.624 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:15:41.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:15:41.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.323969958999999, 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.628 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:15:41.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:15:41.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.322959569999998, 2024-09-21T15:15:41.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:15:41.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:15:41.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.322099163000001, 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.640 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.320769292000001, 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:15:41.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:15:41.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:15:41.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.319309873000002, 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:41.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:41.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:41.649 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:41.650 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:41.651 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:41.648+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:41.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:41.652 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:41.652 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:41.652 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:41.652 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:41.652 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:41.652 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:41.751 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:41.748+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:41.807 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:41.807 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:41.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:41.852 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:41.850+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:41.922 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:41.922 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:41.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:41.952 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:41.948+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:42.054 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:42.048+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:42.084 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.084 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:42.155 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:42.152+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:42.200 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.200 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:42.256 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:42.252+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:42.267 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:15:42.316 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.316 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:42.357 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:42.352+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:42.435 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.435 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:42.457 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:42.452+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:42.556 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.556 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.558 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:42.556+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:42.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:42.659 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:42.656+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:42.673 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.673 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:42.760 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:42.756+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:42.792 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.792 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.802 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:42.861 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:42.858+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:42.918 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:42.918 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:42.928 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:42.963 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:42.962+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:43.034 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:43.035 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:43.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:43.064 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:43.060+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:43.148 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:43.148 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:43.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:43.164 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:43.160+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:43.266 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:43.260+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:43.269 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:43.269 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:43.279 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:43.345 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:15:43.345 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:15:43.345 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T15:15:43.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:15:43.390 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:43.390+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:43.434 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:43.434 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:43.444 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:43.467 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:43.464+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:43.553 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:43.553 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:43.563 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:43.568 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:43.566+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:43.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:15:43.667 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:43.667 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:43.667 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 26, 2024-09-21T15:15:43.667 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:15:43.667 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:15:43.667 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:15:42.254123+0000", 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:15:05.841121+0000", 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5799936, 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:15:43.668 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 2, 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "jewel", 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:15:43.669 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:15:43.670 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:15:43.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:15:43.672 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:15:43.673 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:15:43.674 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:15:43.676 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 20, 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:15:43.678 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.679 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:15:43.680 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.25, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.25, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2.25, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2.25, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:15:43.681 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:15:43.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:15:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:15:43.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:15:43.691 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:15:43.692 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:15:43.697 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:15:43.702 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 22, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:15:43.708 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:15:43.711 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:15:43.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:15:43.714 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:15:43.715 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:15:43.716 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [], 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [], 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:15:43.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:43.718 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:43.720 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:43.720 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:43.721 INFO:tasks.thrashosds.thrasher:j is {'epoch': 26, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:15:42.254123+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:15:05.841121+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:15:43.721 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T15:15:43.721 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:43.718+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:43.721 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:43.795 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:43.819 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:43.820 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:43.816+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:43.831 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:43.831 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:43.837 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:43.841 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:43.861 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:43.883 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:43.921 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:43.916+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:43.949 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:43.949 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:43.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:43.949 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:43.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.949 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:15:43.949 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.051258666999999, 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:43.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:15:43.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:15:43.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.029094488999998, 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:15:43.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.027648147000001, 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:43.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:15:43.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:15:43.962 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.026674590999999, 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:43.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.966 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.024247730999999, 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:15:43.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.023539916000001, 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.971 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:43.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:15:43.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.951022390999999, 2024-09-21T15:15:43.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:15:43.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.979 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.950229685, 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:15:43.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:15:43.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.949063284000001, 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:15:43.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:15:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:43.988 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.947272423000001, 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:15:43.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.992 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.916656045, 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:43.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.914687834999999, 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:43.997 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:15:43.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:15:44.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.912934816, 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:15:44.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.911550175999999, 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.710701723, 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.010 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:15:44.014 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.70993679, 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:15:44.018 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.708926400999999, 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:44.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.022 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.708065994000002, 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:15:44.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:15:44.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:15:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.706736122999999, 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:44.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:15:44.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:15:44.030 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.705276703999999, 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:15:44.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:15:44.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:15:44.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:15:44.035 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.035 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.035 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.035 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:44.035 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:44.035 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:44.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:44.037 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:44.034+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:44.137 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:44.134+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:44.154 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:44.154 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:44.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:44.238 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:44.238+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:44.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:15:44.322 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:44.322 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:44.322 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:44.322 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:44.322 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:44.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:44.374 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:44.374+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:44.439 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:44.438+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:44.476 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.649116811999999, 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:15:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.648501951, 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:15:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.656646796, 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.651758483000002, 2024-09-21T15:15:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:15:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:15:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.646491864000001, 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:15:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:15:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.64421785, 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:15:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.504 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:44.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:44.554 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:44.554+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:44.613 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:44.613 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:44.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:44.641 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:44.638+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:44.726 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:44.726 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:44.726 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:44.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:44.782 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:44.782+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:44.842 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:44.840+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.067342872000001, 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:15:44.849 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:15:44.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:15:44.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.933532501999998, 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.933473275000001, 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.861 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:15:44.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:15:44.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.819150446999998, 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:15:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.611582586000001, 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:15:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.611412646000002, 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.879 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:15:44.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:15:44.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.611383746000001, 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.885 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:15:44.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.611332270999998, 2024-09-21T15:15:44.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.895 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.611153581, 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:15:44.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:15:44.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.611070773000002, 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:15:44.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:15:44.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.610912917, 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:15:44.906 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:44.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:15:44.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.610788205999999, 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.610758153999999, 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.917 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.922 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.164550224999999, 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:15:44.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:15:44.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.928 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.164509616, 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:15:44.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:15:44.933 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.164458203999999, 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.934 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:15:44.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:15:44.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.164273241, 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.164243709000001, 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.945 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:44.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:15:44.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:15:44.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:15:44.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.164201013, 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:15:44.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:44.956 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.163959341000002, 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:44.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:44.962 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:44.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:44.964 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:44.960+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:45.064 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:45.060+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:45.078 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:45.078 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:45.078 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:45.078 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:45.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:45.165 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:45.160+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:45.195 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:45.196 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:45.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:45.266 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:45.266+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.503206654, 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:15:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:15:45.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:45.327 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:45.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:45.378 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:45.376+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:45.448 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:45.448 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:45.448 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:45.448 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:45.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:45.468 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:45.466+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:45.569 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:45.564+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:45.570 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:45.570 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:45.570 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:45.570 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:45.570 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:45.580 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:45.669 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:45.666+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:45.730 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:45.730 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:45.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:45.730 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:15:45.730 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:45.740 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:45.770 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:45.770+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:45.849 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:45.849 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:45.850 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:45.850 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:45.859 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:45.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:45.870+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:45.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:15:45.965 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:45.965 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:45.965 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:45.965 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:45.965 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:45.971 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:45.968+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:45.974 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:46.072 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:46.072 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:46.068+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:46.079 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.25912348, 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:46.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:46.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.249566896000001, 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:46.084 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:46.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:46.088 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:46.090 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:46.090 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:46.108 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:46.129 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:46.152 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:46.173 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:46.168+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:46.174 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:46.195 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:46.195 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:46.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:46.274 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:46.274+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:46.355 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:46.355 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:46.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:46.375 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:46.372+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:46.467 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:46.467 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:46.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:46.477 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:46.474+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:46.578 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:46.578+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:46.597 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:46.597 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:46.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:46.679 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:46.678+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:46.710 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:46.710 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:46.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:46.779 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:46.776+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:46.828 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:46.828 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:46.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:46.880 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:46.878+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:46.946 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:46.946 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:46.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:46.981 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:46.976+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:47.081 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:47.076+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:47.111 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.111 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:47.182 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:47.176+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:47.231 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.231 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:47.282 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:47.282+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:47.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:15:47.346 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.346 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:47.383 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:47.382+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:47.459 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.459 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:47.484 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:47.480+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:47.574 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.574 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.584 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:47.585 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:47.582+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:47.686 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:47.686+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:47.701 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.701 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.711 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:47.787 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:47.784+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:47.817 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.817 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.827 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:47.888 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:47.886+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:47.931 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:47.931 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:47.941 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:47.989 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:47.984+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:48.051 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:48.051 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:48.060 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:48.090 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:48.090+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:48.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:15:48.191 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:48.190+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:48.212 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:48.212 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:48.222 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:48.292 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:48.290+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:15:48.332 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:48.332 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:48.338 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:48.342 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:48.358 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:48.375 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:48.392 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:48.388+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:48.403 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:48.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:48.444 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:48.444 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:48.454 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:48.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:48.494 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:48.488+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:48.554 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:48.554 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:48.554 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:48.554 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:48.554 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:48.563 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:48.594 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:48.594+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:48.695 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:48.692+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:48.714 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:48.714 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:48.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:48.714 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:48.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.714 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:15:48.714 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.817220720000002, 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.718 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.795056542000001, 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:15:48.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.7936102, 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.723 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.727 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.792636644000002, 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:48.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:15:48.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:15:48.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.790209783999998, 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.789501969, 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.736 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.740 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.716984444000001, 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:15:48.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:15:48.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.716191737999999, 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:48.749 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.715025337, 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.713234476, 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:15:48.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:15:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.682618097999999, 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:15:48.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:15:48.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.680649888000001, 2024-09-21T15:15:48.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.766 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.678896868999999, 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:15:48.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.677512229000001, 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:15:48.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.476663775999999, 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:15:48.775 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:15:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.475898843, 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:15:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.474888453999998, 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:15:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.474028047000001, 2024-09-21T15:15:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:15:48.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.472698176000002, 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:15:48.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.471238756999998, 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:48.797 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:48.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:15:48.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:15:48.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:48.801 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:48.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:15:48.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:15:48.802 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T15:15:48.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-09-21T15:15:48.804 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:48.800+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:48.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:48.904 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:48.900+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:48.955 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:48.955 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:48.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:49.005 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:49.002+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:49.063 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:49.064 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:49.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:49.114 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:49.114+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:49.118 INFO:teuthology.orchestra.run.smithi067.stdout:moved 0 / 18 (0%) 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:avg 3 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 0.816497 -> 0.816497 (expected baseline 1.58114) 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.2 with 2 -> 2 pgs (0.666667 -> 0.666667 * mean) 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.1 with 4 -> 4 pgs (1.33333 -> 1.33333 * mean) 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:oload 1000 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 0.05 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0003 2024-09-21T15:15:49.119 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0033 2024-09-21T15:15:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:15:49.134 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T15:15:49.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-09-21T15:15:49.208 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:49.204+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.39546666, 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:15:49.223 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:49.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:15:49.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:49.227 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.394851799000001, 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:15:49.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.402996644000002, 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:15:49.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.398108331, 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.241 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.392841711999999, 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:49.242 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:15:49.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.390567698000002, 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:15:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.252 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:49.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:49.308 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:49.306+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:49.354 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:49.354 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:49.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:49.410 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:49.410+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:moved 0 / 18 (0%) 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:avg 3 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 0.816497 -> 0.816497 (expected baseline 1.58114) 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.2 with 2 -> 2 pgs (0.666667 -> 0.666667 * mean) 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.1 with 4 -> 4 pgs (1.33333 -> 1.33333 * mean) 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:oload 110 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 3 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:15:49.427 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0003 2024-09-21T15:15:49.428 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0004 2024-09-21T15:15:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:15:49.442 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T15:15:49.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-09-21T15:15:49.510 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:49.510+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:49.518 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:49.518 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:49.518 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:49.518 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:49.518 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:49.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:49.611 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:49.610+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:49.684 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.900968395, 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:15:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:15:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.767158025000001, 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:15:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.767098797999999, 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:15:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:15:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.65277597, 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:15:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:15:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.445208108999999, 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:15:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.445038169, 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:15:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.445009269, 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.444957794, 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:15:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:15:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.444779103999998, 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:15:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:15:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.444696296, 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:15:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.741 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.444538439999999, 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:15:49.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.444413729000001, 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:15:49.747 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.444383677000001, 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:15:49.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:15:49.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.998175748000001, 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:15:49.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:15:49.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.998135138999999, 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:15:49.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:15:49.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.768 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.998083727000001, 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:15:49.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:15:49.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.997898763999999, 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.774 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:15:49.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:15:49.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.997869231999999, 2024-09-21T15:15:49.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:15:49.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.997826536000002, 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:15:49.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:15:49.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:15:49.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:15:49.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.997584864, 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:49.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:49.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:49.796 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:49.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:49.798 INFO:teuthology.orchestra.run.smithi067.stdout:moved 0 / 18 (0%) 2024-09-21T15:15:49.798 INFO:teuthology.orchestra.run.smithi067.stdout:avg 3 2024-09-21T15:15:49.798 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 0.816497 -> 0.816497 (expected baseline 1.58114) 2024-09-21T15:15:49.798 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.2 with 2 -> 2 pgs (0.666667 -> 0.666667 * mean) 2024-09-21T15:15:49.798 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.1 with 4 -> 4 pgs (1.33333 -> 1.33333 * mean) 2024-09-21T15:15:49.798 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:49.799 INFO:teuthology.orchestra.run.smithi067.stdout:oload 1000 2024-09-21T15:15:49.799 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 1 2024-09-21T15:15:49.799 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:15:49.799 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0003 2024-09-21T15:15:49.799 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0033 2024-09-21T15:15:49.800 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:15:49.800 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T15:15:49.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-09-21T15:15:49.801 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:49.792+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:49.898 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:49.896+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:49.951 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:49.951 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:49.952 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:49.952 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:49.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:49.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:49.998+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:50.060 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:50.060 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:50.060 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:50.061 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:50.061 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:50.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:50.114 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:50.114+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:50.153 INFO:teuthology.orchestra.run.smithi067.stdout:moved 0 / 18 (0%) 2024-09-21T15:15:50.153 INFO:teuthology.orchestra.run.smithi067.stdout:avg 3 2024-09-21T15:15:50.153 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 0.816497 -> 0.816497 (expected baseline 1.58114) 2024-09-21T15:15:50.153 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.2 with 2 -> 2 pgs (0.666667 -> 0.666667 * mean) 2024-09-21T15:15:50.153 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.1 with 4 -> 4 pgs (1.33333 -> 1.33333 * mean) 2024-09-21T15:15:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:50.154 INFO:teuthology.orchestra.run.smithi067.stdout:oload 1000 2024-09-21T15:15:50.154 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 1 2024-09-21T15:15:50.154 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:15:50.154 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0003 2024-09-21T15:15:50.154 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0033 2024-09-21T15:15:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:15:50.168 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T15:15:50.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-09-21T15:15:50.201 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:50.196+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.399736723, 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:50.224 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:50.230 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:50.302 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:50.302+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:50.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:15:50.341 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:50.341 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:50.341 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:50.341 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:50.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:50.403 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:50.400+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:50.455 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:50.455 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:50.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:50.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:15:50.500 INFO:teuthology.orchestra.run.smithi067.stdout:moved 0 / 18 (0%) 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:avg 3 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 0.816497 -> 0.816497 (expected baseline 1.58114) 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.2 with 2 -> 2 pgs (0.666667 -> 0.666667 * mean) 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.1 with 4 -> 4 pgs (1.33333 -> 1.33333 * mean) 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:oload 110 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 3 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0003 2024-09-21T15:15:50.501 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0004 2024-09-21T15:15:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:15:50.510 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:50.508+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:50.575 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:50.575 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:50.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:50.575 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:15:50.575 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:50.585 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:50.604 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:50.600+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:50.610 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:50.627 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:50.647 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:50.669 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:50.690 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:50.691 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:50.691 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:50.701 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:50.705 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:50.702+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:50.712 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:50.806 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:50.800+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:50.849 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:50.850 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:50.859 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:50.907 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:50.906+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:50.962 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:50.963 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:50.972 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:51.007 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:51.004+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:51.076 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:51.076 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:51.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:51.127 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:51.124+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:51.209 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:51.210+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:51.235 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:51.235 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:51.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:51.310 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:51.310+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:51.328 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:15:51.354 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:51.354 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:51.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:51.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:51.410+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:51.508 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:51.508 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:51.511 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:51.510+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:51.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:51.613 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:51.608+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:51.676 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:51.677 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:51.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:51.714 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:51.714+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:51.794 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:51.794 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:51.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:51.815 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:51.812+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:51.916 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:51.914+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:51.916 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:51.916 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:51.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:52.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:52.014+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:52.032 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.032 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:52.117 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:52.113+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:52.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:52.217+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:52.263 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.263 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:52.318 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:52.317+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:52.420 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:52.416+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:52.425 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.425 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:52.521 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:52.517+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:52.544 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.544 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:52.622 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:52.620+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:52.672 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.672 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.682 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:52.723 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:52.721+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:52.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:15:52.791 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.791 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.802 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:52.824 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:52.821+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:52.882 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:52.905 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:52.910 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:52.911 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:52.920 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:52.925 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:52.921+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:52.927 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:52.950 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:52.972 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:52.992 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:53.031 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:53.031 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:53.031 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:53.031 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:53.040 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:53.050 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:53.049+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:53.127 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:53.125+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:53.143 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:53.143 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:53.143 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:53.143 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:53.143 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:53.152 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:53.227 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:53.229+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:53.261 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:53.261 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:53.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:53.261 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:53.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 29.440663262000001, 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.265 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.431106677999999, 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.269 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.270 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.270 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:53.271 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:53.329 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:53.325+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:53.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:15:53.375 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:53.375 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:53.375 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:53.375 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:53.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:53.429 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:53.425+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:53.489 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:53.490 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:53.499 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:53.530 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:53.529+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:53.612 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:53.612 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:53.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:53.612 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:53.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.714006777999998, 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:15:53.617 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.691842600000001, 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:15:53.621 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.690396258, 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:15:53.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:53.626 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.689422702000002, 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:15:53.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.630 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.686995842000002, 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:53.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:15:53.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.686288027000003, 2024-09-21T15:15:53.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:15:53.639 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.613770502000001, 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:53.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:15:53.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.612977796000003, 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:53.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:15:53.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:15:53.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.611811394999997, 2024-09-21T15:15:53.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:15:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:15:53.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.652 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.610020534, 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.579404156000003, 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:15:53.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.577435946000001, 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:15:53.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.575682927000003, 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.666 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:15:53.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:53.670 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.574298286999998, 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:53.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:15:53.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.373449833999999, 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.372684901, 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:15:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:15:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.371674511999998, 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:15:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.370814105000001, 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:15:53.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.369484233999998, 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:15:53.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.368024814999998, 2024-09-21T15:15:53.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:53.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:15:53.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:53.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:53.704 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:53.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:53.706 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:53.701+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:53.807 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:53.804+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:53.822 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:53.822 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:53.822 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:53.822 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:53.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:53.908 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:53.904+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:53.938 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:53.938 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:53.938 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:53.938 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:53.938 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:53.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:54.009 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:54.004+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:54.058 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:54.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.231531577999998, 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.059 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:15:54.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:15:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.230916716999999, 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:15:54.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:15:54.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.067 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.239061562, 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:15:54.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:15:54.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:15:54.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.234173249000001, 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.073 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:15:54.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.228906630000001, 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:15:54.077 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:15:54.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.226632616, 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:54.083 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:15:54.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:15:54.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.087 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:54.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:54.110 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:54.109+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:54.200 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:54.201 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:54.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:54.211 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:54.209+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:54.312 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:54.309+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:15:54.366 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:54.366 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:54.367 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:54.367 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:54.367 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:54.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:54.413 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:54.408+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:54.514 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:54.509+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:54.529 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.746116262000001, 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:15:54.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.612305892000002, 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:15:54.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:15:54.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:15:54.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:15:54.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:15:54.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.612246665000001, 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.541 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:15:54.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:15:54.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:15:54.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.497923837000002, 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.547 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.552 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.290355976000001, 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:15:54.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:15:54.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.290186036000001, 2024-09-21T15:15:54.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:15:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:15:54.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.290157135999998, 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:15:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:15:54.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.290105660999998, 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:15:54.569 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.574 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.289926971, 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:15:54.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:15:54.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:15:54.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:15:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.289844162999998, 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.289686306999997, 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:15:54.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.289561595999999, 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.591 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:54.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:15:54.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:15:54.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.289531543999999, 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.597 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:15:54.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:15:54.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.843323615000003, 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:15:54.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.843283006, 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.608 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:54.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:15:54.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.843231594000002, 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.614 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:15:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:15:54.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.843046631, 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:15:54.619 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.843017099000001, 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.625 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:54.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:15:54.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:15:54.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:15:54.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.842974402999999, 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.630 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:15:54.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:15:54.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.842732730999998, 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:54.636 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:15:54.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.642 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:54.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:54.643 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:54.640+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:54.744 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:54.740+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:54.759 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:54.759 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:54.760 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:54.760 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:54.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:54.846 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:54.840+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:54.874 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:54.875 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:54.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:54.945 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:54.941+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:54.992 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.172770663999998, 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:54.997 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:55.002 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:55.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:15:55.047 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:55.044+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:55.120 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:55.120 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:55.120 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:55.120 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:55.131 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:55.148 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:55.144+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:55.190 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:55.219 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:55.234 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:55.234 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:55.235 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:55.235 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:55.235 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:55.240 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:55.244 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:55.248 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:55.244+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:55.262 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:55.308 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:55.311 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:55.349 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:55.344+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:55.363 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:55.363 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:55.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:55.450 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:55.448+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:55.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:15:55.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T15:15:55.524 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:55.524 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:55.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:55.578 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:55.576+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:55.639 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:55.640 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:55.649 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:55.651 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:55.648+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:55.752 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:55.752 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:55.752 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:55.748+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:55.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:55.853 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:55.849+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:55.869 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:55.869 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:55.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:55.954 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:55.948+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:55.987 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:55.987 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:55.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:56.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:56.053+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:56.106 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.107 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:56.156 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:56.152+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:56.236 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.236 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:56.256 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:56.253+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T15:15:56.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:15:56.356 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:56.353+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:56.359 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.359 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T15:15:56.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:56.457 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:56.452+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:56.525 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.525 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:56.557 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:56.552+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:56.636 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.636 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:56.657 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:56.653+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:56.745 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.745 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:56.758 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:56.752+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:56.859 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:56.856+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:56.861 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.861 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:56.960 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:56.957+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:56.975 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:56.975 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:56.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:57.061 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:57.056+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:57.141 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:57.142 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:57.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:57.162 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:57.156+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:57.262 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:57.261+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:57.265 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:57.265 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:57.275 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:57.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:15:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T15:15:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:15:57.363 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:57.360+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:57.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T15:15:57.394 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:57.394 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:57.405 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:57.463 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:57.461+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:57.476 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:57.497 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:57.509 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:57.509 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:57.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:57.519 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:15:57.538 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:57.558 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:57.564 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:57.561+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:57.578 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:57.581 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:57.581 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:57.581 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:57.581 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:57.581 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:57.581 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:57.631 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:57.631 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:57.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:57.631 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:15:57.631 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:57.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:15:57.665 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:57.660+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:57.744 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:57.744 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:57.744 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:57.744 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:57.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:15:57.794 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:57.792+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:57.867 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:57.865+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:57.906 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:57.906 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:57.906 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:57.906 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:57.906 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:57.916 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:15:57.968 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:57.965+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 34.205352091999998, 2024-09-21T15:15:58.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:15:58.030 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.195795508000003, 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.035 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.035 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:58.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:15:58.068 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:58.065+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:58.143 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:58.143 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:58.143 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:15:58.143 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:58.153 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:15:58.169 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:58.165+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:58.262 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:58.262 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:15:58.263 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:15:58.263 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:15:58.263 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:58.270 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:58.269+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:58.272 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:15:58.365 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T15:15:58.371 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:58.369+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:58.381 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.483752967000001, 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:15:58.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.461588788999997, 2024-09-21T15:15:58.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:58.390 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.460142447000003, 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:15:58.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:15:58.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:15:58.394 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.459168890999997, 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:15:58.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:15:58.398 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.456742030999997, 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:15:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.456034215999999, 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.383516690999997, 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:15:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:15:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.382723984999998, 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:15:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.381557583999999, 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:15:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.379766723000003, 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:15:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:15:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:15:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.349150344999998, 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.347182134999997, 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:15:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:15:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:15:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.345429115999998, 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.344044476000001, 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:15:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:15:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.143196023000002, 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:15:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.142431090000002, 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.141420701000001, 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:15:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.140560293999997, 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:15:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.139230423000001, 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:15:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.137771004000001, 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:15:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:15:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:15:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:15:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:15:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:15:58.467 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:15:58.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T15:15:58.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:15:58.510 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:15:58.509+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:15:58.573 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:58.569+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:58.627 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:58.627 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:58.627 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:58.627 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:58.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:15:58.674 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:58.672+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:58.736 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:58.737 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:58.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:15:58.776 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:15:58.772+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:15:58.852 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.025217771999998, 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:58.853 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:15:58.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.024602911000002, 2024-09-21T15:15:58.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:58.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:15:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:15:58.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.032747755999999, 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:58.862 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:15:58.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:15:58.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:15:58.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:15:58.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.027859442999997, 2024-09-21T15:15:58.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:15:58.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:15:58.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:15:58.872 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.022592824, 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:58.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:15:58.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:15:58.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.020318809999999, 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:15:58.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:58.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:58.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:58.883 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:58.883 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:58.884 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:15:58.885 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:58.880+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:58.985 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:58.980+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:58.996 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:58.996 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:58.996 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:58.996 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:59.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:15:59.086 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:59.081+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:59.109 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:59.109 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:59.109 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:59.110 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:59.110 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:59.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:15:59.186 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:15:59.185+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:15:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:59.279 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:15:59.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:15:59.279 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:15:59.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.279 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.279 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.495472360999997, 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:15:59.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:15:59.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:15:59.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:59.285 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.361661990999998, 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.290 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.361602763999997, 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:15:59.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.296 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.247279935999998, 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.039712074999997, 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:15:59.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:15:59.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.039542134999998, 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:15:59.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:15:59.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:15:59.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:15:59.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.313 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.039513235000001, 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.039461760000002, 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.319 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:15:59.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.039283070000003, 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:15:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.329 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.039200262000001, 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:15:59.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:15:59.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:15:59.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.039042406, 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:15:59.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:15:59.340 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.038917695000002, 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.341 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:15:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.038887643000002, 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:15:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:15:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.592679713999999, 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:15:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:15:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.592639105000003, 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:15:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:15:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:15:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.592587692999999, 2024-09-21T15:15:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:15:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:15:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.592402730000003, 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:15:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.592373197999997, 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:15:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:15:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.592330502000003, 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:15:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:15:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:15:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.592088830000002, 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:15:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:15:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:15:59.392 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:59.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:15:59.394 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:59.388+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:59.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:15:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T15:15:59.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T15:15:59.494 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:59.492+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:15:59.559 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:59.559 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:15:59.560 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:59.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:15:59.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:15:59.595 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:59.593+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:59.695 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:59.692+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:59.718 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:15:59.718 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:15:59.719 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:15:59.719 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:15:59.719 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:15:59.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:15:59.769 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:15:59.790 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:15:59.796 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:15:59.793+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:15:59.808 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:15:59.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:15:59.838 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:59.839 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:59.849 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:15:59.853 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:15:59.876 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:15:59.878 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:15:59.879 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:15:59.879 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:15:59.879 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:15:59.879 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:15:59.879 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:15:59.897 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:15:59.892+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:15:59.964 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:15:59.965 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:15:59.975 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:15:59.998 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:15:59.996+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:00.081 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:00.081 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:00.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:00.099 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:00.096+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:00.200 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:00.196+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:00.201 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:00.201 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:00.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:00.301 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:00.296+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:00.366 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:00.366 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:00.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T15:16:00.385 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:16:00.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T15:16:00.418 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:00.416+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:00.475 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:00.475 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:00.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:00.502 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:00.501+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:00.594 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:00.594 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:00.602 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:00.600+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:00.604 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:00.704 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:00.700+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:00.758 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:00.758 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:00.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:00.805 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:00.800+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:00.870 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:00.870 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:00.881 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:00.906 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:00.905+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:01.007 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:01.005+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:01.032 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.032 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:01.107 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:01.104+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:01.158 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.158 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:01.208 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:01.204+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:01.276 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.276 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:01.309 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:01.304+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:01.386 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T15:16:01.397 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.398 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.399 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T15:16:01.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:16:01.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:01.409 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:01.405+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:01.511 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.511 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.512 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:01.508+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:01.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:01.612 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:01.609+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:01.623 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.623 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:01.692 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:16:01.707 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-21T15:16:01.378190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":188736,"kb_used_data":2028,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566042304,"statfs":{"total":579820584960,"available":579627319296,"internally_reserved":0,"allocated":2076672,"data_stored":1860214,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028271"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T15:15:54.335886+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:15:54.335886+0000","last_peered":"2024-09-21T15:15:54.335886+0000","last_clean":"2024-09-21T15:15:54.335886+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:15:54.335886+0000","last_undegraded":"2024-09-21T15:15:54.335886+0000","last_fullsized":"2024-09-21T15:15:54.335886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T15:15:58.533486+0000","last_change":"2024-09-21T15:15:24.826574+0000","last_active":"2024-09-21T15:15:58.533486+0000","last_peered":"2024-09-21T15:15:58.533486+0000","last_clean":"2024-09-21T15:15:58.533486+0000","last_became_active":"2024-09-21T15:15:22.824152+0000","last_became_peered":"2024-09-21T15:15:22.824152+0000","last_unstale":"2024-09-21T15:15:58.533486+0000","last_undegraded":"2024-09-21T15:15:58.533486+0000","last_fullsized":"2024-09-21T15:15:58.533486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:54.629041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066240600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T15:15:54.335946+0000","last_change":"2024-09-21T15:15:24.829674+0000","last_active":"2024-09-21T15:15:54.335946+0000","last_peered":"2024-09-21T15:15:54.335946+0000","last_clean":"2024-09-21T15:15:54.335946+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:15:54.335946+0000","last_undegraded":"2024-09-21T15:15:54.335946+0000","last_fullsized":"2024-09-21T15:15:54.335946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:48.076212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:15:51.488836+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:15:51.488836+0000","last_peered":"2024-09-21T15:15:51.488836+0000","last_clean":"2024-09-21T15:15:51.488836+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:15:51.488836+0000","last_undegraded":"2024-09-21T15:15:51.488836+0000","last_fullsized":"2024-09-21T15:15:51.488836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:15:51.491898+0000","last_change":"2024-09-21T15:15:24.830325+0000","last_active":"2024-09-21T15:15:51.491898+0000","last_peered":"2024-09-21T15:15:51.491898+0000","last_clean":"2024-09-21T15:15:51.491898+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:15:51.491898+0000","last_undegraded":"2024-09-21T15:15:51.491898+0000","last_fullsized":"2024-09-21T15:15:51.491898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:55:21.434241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:15:51.488860+0000","last_change":"2024-09-21T15:15:34.263162+0000","last_active":"2024-09-21T15:15:51.488860+0000","last_peered":"2024-09-21T15:15:51.488860+0000","last_clean":"2024-09-21T15:15:51.488860+0000","last_became_active":"2024-09-21T15:15:34.262327+0000","last_became_peered":"2024-09-21T15:15:34.262327+0000","last_unstale":"2024-09-21T15:15:51.488860+0000","last_undegraded":"2024-09-21T15:15:51.488860+0000","last_fullsized":"2024-09-21T15:15:51.488860+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:15.380455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T15:15:58.362819+0000","last_change":"2024-09-21T15:15:25.660255+0000","last_active":"2024-09-21T15:15:58.362819+0000","last_peered":"2024-09-21T15:15:58.362819+0000","last_clean":"2024-09-21T15:15:58.362819+0000","last_became_active":"2024-09-21T15:15:22.821654+0000","last_became_peered":"2024-09-21T15:15:22.821654+0000","last_unstale":"2024-09-21T15:15:58.362819+0000","last_undegraded":"2024-09-21T15:15:58.362819+0000","last_fullsized":"2024-09-21T15:15:58.362819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:55.119052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000470638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T15:15:54.335726+0000","last_change":"2024-09-21T15:15:24.829906+0000","last_active":"2024-09-21T15:15:54.335726+0000","last_peered":"2024-09-21T15:15:54.335726+0000","last_clean":"2024-09-21T15:15:54.335726+0000","last_became_active":"2024-09-21T15:15:22.819834+0000","last_became_peered":"2024-09-21T15:15:22.819834+0000","last_unstale":"2024-09-21T15:15:54.335726+0000","last_undegraded":"2024-09-21T15:15:54.335726+0000","last_fullsized":"2024-09-21T15:15:54.335726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:32.673395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T15:15:54.335682+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:15:54.335682+0000","last_peered":"2024-09-21T15:15:54.335682+0000","last_clean":"2024-09-21T15:15:54.335682+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:15:54.335682+0000","last_undegraded":"2024-09-21T15:15:54.335682+0000","last_fullsized":"2024-09-21T15:15:54.335682+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":368640,"data_stored":698267,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":311296,"data_stored":107899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":368640,"data_stored":698267,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":315392,"data_stored":107918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607566,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":352256,"data_stored":121652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":2,"up_from":12,"seq":51539607564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":360448,"data_stored":126211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:16:01.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:16:01.713 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:01.708+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:01.748 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.748 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:01.813 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:01.808+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:01.863 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:01.863 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:01.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:01.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:01.914 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:01.912+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:01.998 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:02.001 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:16:02.005 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:02.005 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:02.014 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-21T15:16:01.926670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":189016,"kb_used_data":2280,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566042024,"statfs":{"total":579820584960,"available":579627032576,"internally_reserved":0,"allocated":2334720,"data_stored":1956352,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001290"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380460+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:16:00.380460+0000","last_peered":"2024-09-21T15:16:00.380460+0000","last_clean":"2024-09-21T15:16:00.380460+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:16:00.380460+0000","last_undegraded":"2024-09-21T15:16:00.380460+0000","last_fullsized":"2024-09-21T15:16:00.380460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T15:15:58.533486+0000","last_change":"2024-09-21T15:15:24.826574+0000","last_active":"2024-09-21T15:15:58.533486+0000","last_peered":"2024-09-21T15:15:58.533486+0000","last_clean":"2024-09-21T15:15:58.533486+0000","last_became_active":"2024-09-21T15:15:22.824152+0000","last_became_peered":"2024-09-21T15:15:22.824152+0000","last_unstale":"2024-09-21T15:15:58.533486+0000","last_undegraded":"2024-09-21T15:15:58.533486+0000","last_fullsized":"2024-09-21T15:15:58.533486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:54.629041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066240600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380434+0000","last_change":"2024-09-21T15:15:24.829674+0000","last_active":"2024-09-21T15:16:00.380434+0000","last_peered":"2024-09-21T15:16:00.380434+0000","last_clean":"2024-09-21T15:16:00.380434+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:16:00.380434+0000","last_undegraded":"2024-09-21T15:16:00.380434+0000","last_fullsized":"2024-09-21T15:16:00.380434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:48.076212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.639398+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:16:00.639398+0000","last_peered":"2024-09-21T15:16:00.639398+0000","last_clean":"2024-09-21T15:16:00.639398+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:16:00.639398+0000","last_undegraded":"2024-09-21T15:16:00.639398+0000","last_fullsized":"2024-09-21T15:16:00.639398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":31,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T15:15:59.866702+0000","last_change":"2024-09-21T15:15:24.830325+0000","last_active":"2024-09-21T15:15:59.866702+0000","last_peered":"2024-09-21T15:15:59.866702+0000","last_clean":"2024-09-21T15:15:59.866702+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:15:59.866702+0000","last_undegraded":"2024-09-21T15:15:59.866702+0000","last_fullsized":"2024-09-21T15:15:59.866702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:55:21.434241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.639341+0000","last_change":"2024-09-21T15:15:34.263162+0000","last_active":"2024-09-21T15:16:00.639341+0000","last_peered":"2024-09-21T15:16:00.639341+0000","last_clean":"2024-09-21T15:16:00.639341+0000","last_became_active":"2024-09-21T15:15:34.262327+0000","last_became_peered":"2024-09-21T15:15:34.262327+0000","last_unstale":"2024-09-21T15:16:00.639341+0000","last_undegraded":"2024-09-21T15:16:00.639341+0000","last_fullsized":"2024-09-21T15:16:00.639341+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:15.380455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T15:15:58.362819+0000","last_change":"2024-09-21T15:15:25.660255+0000","last_active":"2024-09-21T15:15:58.362819+0000","last_peered":"2024-09-21T15:15:58.362819+0000","last_clean":"2024-09-21T15:15:58.362819+0000","last_became_active":"2024-09-21T15:15:22.821654+0000","last_became_peered":"2024-09-21T15:15:22.821654+0000","last_unstale":"2024-09-21T15:15:58.362819+0000","last_undegraded":"2024-09-21T15:15:58.362819+0000","last_fullsized":"2024-09-21T15:15:58.362819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:55.119052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000470638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380405+0000","last_change":"2024-09-21T15:15:24.829906+0000","last_active":"2024-09-21T15:16:00.380405+0000","last_peered":"2024-09-21T15:16:00.380405+0000","last_clean":"2024-09-21T15:16:00.380405+0000","last_became_active":"2024-09-21T15:15:22.819834+0000","last_became_peered":"2024-09-21T15:15:22.819834+0000","last_unstale":"2024-09-21T15:16:00.380405+0000","last_undegraded":"2024-09-21T15:16:00.380405+0000","last_fullsized":"2024-09-21T15:16:00.380405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:32.673395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380330+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:16:00.380330+0000","last_peered":"2024-09-21T15:16:00.380330+0000","last_clean":"2024-09-21T15:16:00.380330+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:16:00.380330+0000","last_undegraded":"2024-09-21T15:16:00.380330+0000","last_fullsized":"2024-09-21T15:16:00.380330+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":430080,"data_stored":721157,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":385024,"data_stored":135367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":430080,"data_stored":721157,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":376832,"data_stored":130808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607566,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":352256,"data_stored":121652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":2,"up_from":12,"seq":51539607564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":360448,"data_stored":126211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:16:02.015 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T15:16:02.015 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:02.016 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T15:16:02.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:16:02.016 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:02.012+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:02.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:02.035 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:02.063 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:02.080 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:02.101 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:02.115 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:02.112+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:02.122 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.302298047000001, 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:02.127 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:02.134 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:02.147 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:02.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:02.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:02.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:02.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:02.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:02.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:02.216 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:02.212+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:02.248 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:02.248 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:02.248 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:02.248 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:02.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:02.303 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 36, 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:16:01.377035+0000", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:15:05.841121+0000", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5806080, 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:16:02.304 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 2, 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "jewel", 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:16:02.305 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:16:02.306 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:16:02.307 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:16:02.308 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:16:02.309 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:16:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:16:02.311 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:16:02.312 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:16:02.313 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 20, 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:16:02.314 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:16:02.315 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:16:02.316 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.25, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.25, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2.25, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2.25, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:02.318 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:16:02.319 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.321 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:02.324 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.325 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.326 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:02.327 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:16:02.328 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:02.329 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:16:02.331 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.333 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:02.334 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.335 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.336 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.337 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.338 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:16:02.339 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:02.340 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:16:02.341 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:16:02.342 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:16:02.343 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.344 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:02.345 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 22, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.348 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:02.349 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.350 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:02.351 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:16:02.352 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:02.353 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:02.354 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [], 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [], 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:16:02.355 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:16:02.356 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:16:02.357 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:02.357 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:02.358 INFO:tasks.thrashosds.thrasher:j is {'epoch': 36, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:16:01.377035+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:15:05.841121+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:16:02.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:16:02.359 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:02.356+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:02.369 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:02.369 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:02.370 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:02.370 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:02.380 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:02.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:16:02.459 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:02.456+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:02.482 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:16:02.482 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:02.492 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:02.560 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:02.557+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:02.648 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:02.648 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:02.648 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:02.648 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:02.658 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:02.661 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:02.657+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:02.666 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:16:02.680 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-21T15:16:01.926670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":189016,"kb_used_data":2280,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":566042024,"statfs":{"total":579820584960,"available":579627032576,"internally_reserved":0,"allocated":2334720,"data_stored":1956352,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001290"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380460+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:16:00.380460+0000","last_peered":"2024-09-21T15:16:00.380460+0000","last_clean":"2024-09-21T15:16:00.380460+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:16:00.380460+0000","last_undegraded":"2024-09-21T15:16:00.380460+0000","last_fullsized":"2024-09-21T15:16:00.380460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T15:15:58.533486+0000","last_change":"2024-09-21T15:15:24.826574+0000","last_active":"2024-09-21T15:15:58.533486+0000","last_peered":"2024-09-21T15:15:58.533486+0000","last_clean":"2024-09-21T15:15:58.533486+0000","last_became_active":"2024-09-21T15:15:22.824152+0000","last_became_peered":"2024-09-21T15:15:22.824152+0000","last_unstale":"2024-09-21T15:15:58.533486+0000","last_undegraded":"2024-09-21T15:15:58.533486+0000","last_fullsized":"2024-09-21T15:15:58.533486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:31:54.629041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066240600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":32,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380434+0000","last_change":"2024-09-21T15:15:24.829674+0000","last_active":"2024-09-21T15:16:00.380434+0000","last_peered":"2024-09-21T15:16:00.380434+0000","last_clean":"2024-09-21T15:16:00.380434+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:16:00.380434+0000","last_undegraded":"2024-09-21T15:16:00.380434+0000","last_fullsized":"2024-09-21T15:16:00.380434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:48.076212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.639398+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:16:00.639398+0000","last_peered":"2024-09-21T15:16:00.639398+0000","last_clean":"2024-09-21T15:16:00.639398+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:16:00.639398+0000","last_undegraded":"2024-09-21T15:16:00.639398+0000","last_fullsized":"2024-09-21T15:16:00.639398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":31,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T15:15:59.866702+0000","last_change":"2024-09-21T15:15:24.830325+0000","last_active":"2024-09-21T15:15:59.866702+0000","last_peered":"2024-09-21T15:15:59.866702+0000","last_clean":"2024-09-21T15:15:59.866702+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:15:59.866702+0000","last_undegraded":"2024-09-21T15:15:59.866702+0000","last_fullsized":"2024-09-21T15:15:59.866702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:55:21.434241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.639341+0000","last_change":"2024-09-21T15:15:34.263162+0000","last_active":"2024-09-21T15:16:00.639341+0000","last_peered":"2024-09-21T15:16:00.639341+0000","last_clean":"2024-09-21T15:16:00.639341+0000","last_became_active":"2024-09-21T15:15:34.262327+0000","last_became_peered":"2024-09-21T15:15:34.262327+0000","last_unstale":"2024-09-21T15:16:00.639341+0000","last_undegraded":"2024-09-21T15:16:00.639341+0000","last_fullsized":"2024-09-21T15:16:00.639341+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:15.380455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T15:15:58.362819+0000","last_change":"2024-09-21T15:15:25.660255+0000","last_active":"2024-09-21T15:15:58.362819+0000","last_peered":"2024-09-21T15:15:58.362819+0000","last_clean":"2024-09-21T15:15:58.362819+0000","last_became_active":"2024-09-21T15:15:22.821654+0000","last_became_peered":"2024-09-21T15:15:22.821654+0000","last_unstale":"2024-09-21T15:15:58.362819+0000","last_undegraded":"2024-09-21T15:15:58.362819+0000","last_fullsized":"2024-09-21T15:15:58.362819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:55.119052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000470638,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":33,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380405+0000","last_change":"2024-09-21T15:15:24.829906+0000","last_active":"2024-09-21T15:16:00.380405+0000","last_peered":"2024-09-21T15:16:00.380405+0000","last_clean":"2024-09-21T15:16:00.380405+0000","last_became_active":"2024-09-21T15:15:22.819834+0000","last_became_peered":"2024-09-21T15:15:22.819834+0000","last_unstale":"2024-09-21T15:16:00.380405+0000","last_undegraded":"2024-09-21T15:16:00.380405+0000","last_fullsized":"2024-09-21T15:16:00.380405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:32.673395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":180,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-21T15:16:00.380330+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:16:00.380330+0000","last_peered":"2024-09-21T15:16:00.380330+0000","last_clean":"2024-09-21T15:16:00.380330+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:16:00.380330+0000","last_undegraded":"2024-09-21T15:16:00.380330+0000","last_fullsized":"2024-09-21T15:16:00.380330+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":430080,"data_stored":721157,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31504,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340336,"statfs":{"total":96636764160,"available":96604504064,"internally_reserved":0,"allocated":385024,"data_stored":135367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":430080,"data_stored":721157,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31492,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340348,"statfs":{"total":96636764160,"available":96604516352,"internally_reserved":0,"allocated":376832,"data_stored":130808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607566,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":352256,"data_stored":121652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":2,"up_from":12,"seq":51539607564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":360448,"data_stored":126211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:16:02.681 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [0, 1, 4, 3] 2024-09-21T15:16:02.681 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.4', '0', '1', '4', '3'] 2024-09-21T15:16:02.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.4 0 1 4 3 2024-09-21T15:16:02.761 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:02.756+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:02.767 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:02.767 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:02.767 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:02.767 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:02.767 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:02.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:02.862 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:02.860+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:02.928 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.108513692000002, 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:02.929 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:02.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:02.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:02.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 38.098957108, 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:02.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:02.938 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:02.939 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:02.963 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:02.960+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:03.046 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:03.047 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:03.056 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:03.098 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:03.096+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:03.164 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:03.165+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:03.208 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:03.208 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:03.208 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:03.208 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:03.208 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:03.217 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:03.265 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:03.261+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.427337203, 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:03.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:03.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.405173025000003, 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.403726683000002, 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:03.334 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:03.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:03.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:03.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.402753126999997, 2024-09-21T15:16:03.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:03.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.400326266999997, 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:03.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:03.346 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.399618451999999, 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.327100926999996, 2024-09-21T15:16:03.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:03.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:03.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.326308220999998, 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:03.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:03.359 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.325141819999999, 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:03.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:03.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.323350959000003, 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.364 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:03.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:03.368 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.292734580999998, 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:03.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:03.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.290766370999997, 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.376 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.289013351999998, 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.380 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.287628712, 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:03.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:03.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.086780259000001, 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:03.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.086015326000002, 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.389 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:03.393 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.085004937000001, 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.397 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.084144530000003, 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:03.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.082814659, 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.402 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:03.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:03.406 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.081355240000001, 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:03.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:03.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:03.410 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.411 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:03.411 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:03.411 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:set 2.4 pg_upmap_items mapping to [0->1,4->3] 2024-09-21T15:16:03.412 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:16:03.413 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:03.408+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:03.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:03.513 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:03.509+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:03.516 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:03.516 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:03.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:03.614 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:03.612+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:03.636 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:03.637 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:03.637 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:03.637 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:03.637 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:03.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:03.715 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:03.712+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.930298428999997, 2024-09-21T15:16:03.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:16:03.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:16:03.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.929683568000002, 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:16:03.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.937828412999998, 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:03.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:03.771 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.932940100000003, 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.927673480999999, 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:03.775 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:03.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.780 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.925399466999998, 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:03.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:16:03.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:16:03.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:03.784 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:03.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:03.830 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:03.829+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:03.896 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:03.896 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:03.897 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:03.897 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:03.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:03.918 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:03.916+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:04.013 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:04.014 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:04.018 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:04.016+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:04.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:04.119 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:04.116+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:04.138 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:04.138 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:04.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:04.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.353835418000003, 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.139 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:04.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:04.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:04.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:04.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:04.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:04.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:04.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:04.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:04.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:04.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:04.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:04.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:04.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:04.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:04.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.220025047999997, 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.149 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:04.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:04.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.219965821000002, 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.155 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:04.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.160 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.105642992999996, 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:04.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:04.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:04.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:04.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:04.168 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.168 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.168 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.168 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.168 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:04.169 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.898075132000002, 2024-09-21T15:16:04.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:04.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.169 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.169 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.169 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.169 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:04.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:04.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:04.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:04.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:04.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:04.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:04.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:04.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:04.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:04.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:04.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:04.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:04.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:04.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:04.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:04.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:04.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:04.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:04.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.181 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.181 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:04.181 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897905192000003, 2024-09-21T15:16:04.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:04.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.182 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.182 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.182 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.182 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.182 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:04.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.182 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:04.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897876291999999, 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:04.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:04.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:04.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:04.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897824817, 2024-09-21T15:16:04.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:04.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:04.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:04.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897646127000002, 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.200 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:04.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:04.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:04.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897563319, 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.239 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.245 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897405462999998, 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.251 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.253 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.253 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897280752, 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:04.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.897250700000001, 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:04.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:04.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:04.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.451042770999997, 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.265 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:04.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.270 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.451002162000002, 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:04.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:04.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.450950749999997, 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:04.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:04.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:04.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.450765787000002, 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.282 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.450736255000002, 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.450693559000001, 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.450451887, 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:04.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:04.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:04.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:04.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:04.304 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:04.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:04.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:04.307 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:04.305+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:04.407 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:04.405+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:04.466 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:04.467 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:04.467 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:04.467 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:04.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:04.498 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:04.507 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:04.505+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:04.540 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:04.564 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:04.588 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:04.589 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:04.591 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:04.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:04.608 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:04.604+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:04.614 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:04.709 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:04.704+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:04.720 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:04.720 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:04.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:04.810 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:04.809+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:04.851 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:04.851 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:04.861 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:04.911 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:04.909+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:04.967 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:04.967 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:04.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:05.012 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:05.008+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:05.085 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:05.085 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:05.094 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:05.138 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:05.137+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:05.214 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:05.213+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:05.247 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:05.247 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:05.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:05.315 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:05.313+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:05.367 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:05.367 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:05.376 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:05.418 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:05.417+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:05.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:16:05.517 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:05.513+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:05.526 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:05.526 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:05.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:05.618 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:05.617+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:05.644 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:05.645 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:05.654 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:05.719 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:05.717+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:05.763 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:05.764 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:05.773 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:05.820 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:05.821+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:05.878 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:05.879 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:05.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:05.934 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:05.933+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:06.004 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:06.004 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:06.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:06.022 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:06.021+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:06.123 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:06.121+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:06.123 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:06.123 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:06.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:06.224 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:06.221+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:06.283 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:06.283 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:06.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:06.324 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:06.321+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:06.399 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:06.399 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:06.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:06.420 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:16:06.425 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:06.421+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:06.525 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:06.525 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:06.526 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:06.525+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:06.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:06.627 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:06.625+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:06.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:06.643 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:06.644 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:06.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:06.728 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:06.725+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:06.774 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:06.774 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:06.776 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:06.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:06.795 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:06.815 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:06.829 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:06.825+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:06.838 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:06.863 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:06.887 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:06.890 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:06.896 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:06.897 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:06.897 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:06.897 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:06.897 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:06.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:06.930 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:06.929+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:07.014 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:07.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:07.014 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:07.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.014 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.194098816, 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:07.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:07.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:07.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:07.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:07.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:07.019 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:07.024 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:07.070 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:07.069+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:07.131 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:07.129+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:07.141 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:07.141 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:07.141 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:07.141 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:07.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:07.232 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:07.229+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:07.254 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:07.254 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:07.254 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:07.254 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:07.254 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:07.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:07.333 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:07.333+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:07.412 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:07.412 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:07.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:07.412 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:16:07.412 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:07.422 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:07.433 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:07.433+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:07.526 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:07.527 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:07.527 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:07.527 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:07.534 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:07.533+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:07.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:07.636 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:07.637+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:07.645 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:07.645 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:07.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:07.737 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:07.733+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:07.765 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:07.765 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 43.945904478999999, 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:07.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:07.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:07.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:07.769 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.936347894999997, 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:07.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:07.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:07.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:07.774 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:07.774 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:07.774 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:07.774 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:07.774 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:07.774 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:07.775 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:07.837 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:07.833+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:07.882 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:07.882 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:07.882 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:07.882 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:07.892 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:07.938 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:07.937+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:08.002 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:08.003 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:08.003 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:08.003 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:08.003 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:08.012 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:08.038 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:08.037+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.222640576000003, 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.200476397999999, 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.199030055999998, 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.1980565, 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.19562964, 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.194921825000002, 2024-09-21T15:16:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.122404299999999, 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.121611594000001, 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.120445193000002, 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.118654331999998, 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.088037954000001, 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.086069744, 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.084316725000001, 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:08.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:08.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:08.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.082932085000003, 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:08.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:08.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.178 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.882083631999997, 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:08.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:08.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:08.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:08.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.881318698999998, 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:08.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:08.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:08.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:08.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.880308309999997, 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.879447902999999, 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.191 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:08.195 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.878118032000003, 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:08.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:08.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.876658613000004, 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.203 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.204 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.204 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:08.204 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:08.204 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:08.205 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:08.201+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:08.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:08.306 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:08.305+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:08.326 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:08.326 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:08.326 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:08.326 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:08.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:08.407 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:08.405+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:08.423 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:08.423 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:08.423 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T15:16:08.423 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:16:08.423 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:16:08.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-21T15:16:08.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:16:08.449 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:08.450 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:08.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:08.507 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:08.505+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:08.539 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:08.540 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "" 2024-09-21T15:16:08.540 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:08.609 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:08.605+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:08.613 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.785994015999997, 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:16:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.785379155000001, 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:16:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:16:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.793523999999998, 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.788635687000003, 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:16:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.783369067999999, 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.781095053999998, 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:16:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:16:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:16:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:08.641 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:08.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:08.710 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:08.709+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:08.795 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:08.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:08.795 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:08.795 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:08.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:08.811 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:08.809+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:08.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:08.911 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:08.909+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:08.912 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:08.912 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:08.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:09.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:09.009+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:09.033 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:09.033 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:09.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:09.033 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:09.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.250265853999998, 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:09.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.039 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.116455483999999, 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.040 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.116396256999998, 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.045 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.050 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.002073428999999, 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:09.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.794505567999998, 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.794335627999999, 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:09.062 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:09.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:09.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:09.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.794306728000002, 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:09.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.794255253000003, 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:09.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:09.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:09.078 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.794076562999997, 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:09.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:09.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.793993755000002, 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:09.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:09.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.793835899000001, 2024-09-21T15:16:09.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:09.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:09.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.793711188000003, 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:09.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:09.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:09.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.793681136000004, 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:09.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:09.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:09.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.347473207, 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.106 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:09.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.347432597999997, 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:09.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:09.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:09.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.347381186, 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.117 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.122 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.347196222999997, 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.123 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:09.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:09.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.128 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.347166690999998, 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.129 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.134 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.347123994999997, 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:09.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.346882323000003, 2024-09-21T15:16:09.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:09.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:09.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:09.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:09.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:09.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:09.146 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:09.147 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:09.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:09.148 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:09.149 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:09.149 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:09.149 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:09.149 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:09.149 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:09.145+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:09.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:09.249 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:09.245+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:09.262 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:09.262 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:09.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:09.349 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:09.345+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:09.382 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:09.382 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:09.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:16:09.450 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:09.449+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:09.500 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:09.500 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:09.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:09.551 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:09.549+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:09.626 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:09.626 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:09.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:09.651 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:09.649+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:09.747 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:09.747 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:09.753 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:09.749+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:09.757 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:09.853 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:09.853+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:09.907 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:09.907 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:09.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:09.954 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:09.953+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:10.027 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.027 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:10.082 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:10.081+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:10.156 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:10.153+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:10.189 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.189 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.199 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:10.257 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:10.253+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:10.307 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.307 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:10.357 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:10.357+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:10.458 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:10.457+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:10.467 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.467 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:10.559 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:10.557+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:10.579 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.579 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.589 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:10.660 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:10.657+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:10.737 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.738 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:10.794 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:10.793+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:10.862 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:10.861+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:10.867 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.867 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:10.963 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:10.961+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:10.986 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:10.987 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:10.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:11.064 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:11.061+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:11.105 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:11.105 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:11.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:11.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:11.165 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:11.161+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:11.223 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:11.224 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:11.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:11.266 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:11.261+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:11.300 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:11.322 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:11.341 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:11.341 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:11.341 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:11.346 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:11.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:11.366 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:11.365+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:11.368 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:11.391 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:11.421 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:11.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.680173513, 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.546363143000001, 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.546303915999999, 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.431981088000001, 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.224413226999999, 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.224243287, 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.224214387000004, 2024-09-21T15:16:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.224162911999997, 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.223984221999999, 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.223901413999997, 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.223743558000002, 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.223618846999997, 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.223588794999998, 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.777380866000001, 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.777340256999999, 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.777288845000001, 2024-09-21T15:16:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.777103881999999, 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.777074349999999, 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.563 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.777031653999998, 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:11.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.568 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.776789981999997, 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.569 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:11.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:11.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:11.574 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.574 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.574 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.574 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:11.574 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:11.574 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:11.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:11.576 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:11.573+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:11.676 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:11.673+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:11.722 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:11.722 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:11.722 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:11.722 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:11.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:11.777 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:11.773+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:11.839 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:11.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:11.878 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:11.877+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:11.979 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:11.977+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:11.998 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:11.998 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:11.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:11.998 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:11.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.178482354000003, 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:11.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:12.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:12.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:12.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:12.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:12.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:12.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:12.003 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:12.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:12.080 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:12.077+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:12.124 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:12.125 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:12.125 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:12.125 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:12.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:12.180 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:12.181+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:12.238 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:12.238 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:12.238 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:12.238 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:12.238 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:12.248 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:12.281 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:12.277+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:12.357 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:16:12.357 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:12.367 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:12.382 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:12.381+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:12.450 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:16:12.477 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:12.478 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:12.478 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:12.478 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:12.483 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:12.481+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:12.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:12.584 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:12.581+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:12.637 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:12.637 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:12.638 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:12.638 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:12.638 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:12.647 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:12.684 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:12.681+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:12.756 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.936699449000002, 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.927142865, 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:12.766 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:12.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:12.785 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:12.785+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:12.886 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:12.885+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:12.921 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:12.921 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:12.922 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:12.922 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:12.931 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:12.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:12.989+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:13.039 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:13.039 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:13.039 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:13.039 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:13.039 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:13.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:13.088 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:13.085+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.261350331000003, 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.157 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:13.161 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.239186152999999, 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.237739810999997, 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.236766254999999, 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.170 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:13.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.234339394999999, 2024-09-21T15:16:13.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:13.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.178 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.233631580000001, 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.161114054999999, 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.186 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.160321349, 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:13.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:13.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.159154948000001, 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.191 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:13.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:13.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:13.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:13.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.157364086999998, 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.196 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:13.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:13.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:13.200 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.126747709, 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:13.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:13.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.124779498999999, 2024-09-21T15:16:13.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.12302648, 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.121641840000002, 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.920793387000003, 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.920028453999997, 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.919018065000003, 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.918157657999998, 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.916827787000003, 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.915368368000003, 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:13.244 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:13.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:13.247 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:13.245+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:13.347 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:13.345+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:13.364 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:13.364 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:13.364 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:13.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:13.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:13.447 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:13.445+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:13.486 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:13.486 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:13.486 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:13.486 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:13.487 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:13.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:13.548 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:13.545+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:13.551 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:13.551 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:13.551 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-21T15:16:13.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-21T15:16:13.576 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:13.599 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:13.599 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:13.599 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:13.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:13.617 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:13.639 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:13.650 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:13.649+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:13.663 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:13.687 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:13.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-21T15:16:13.721 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:13.721 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:13.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:13.774 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:13.773+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:13.851 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:13.849+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:13.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-21T15:16:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:13.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:13.952 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:13.949+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:13.993 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:13.993 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:14.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-21T15:16:14.053 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:14.053+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:14.107 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:14.107 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:14.154 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:14.153+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:14.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-21T15:16:14.219 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:14.219 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:14.274 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:14.273+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:14.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-21T15:16:14.356 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:14.353+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:14.417 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:14.417 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.427 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:14.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:16:14.473 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:14.473+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:14.543 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:14.543 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.553 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:14.556 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:14.553+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:14.655 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:14.655 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.657 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:14.657+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:14.664 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:14.758 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:14.757+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:14.773 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:14.773 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:14.859 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:14.857+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:14.934 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:14.935 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:14.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:14.959 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:14.957+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:15.051 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:15.051 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:15.061 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:15.061 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:15.061+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:15.162 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:15.161+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:15.215 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:15.215 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:15.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:15.263 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:15.261+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:15.333 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:15.333 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:15.343 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:15.364 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:15.361+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:15.448 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:15.448 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:15.457 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:15.465 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:15.461+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:15.473 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:16:15.566 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:15.565+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:15.610 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:15.610 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:15.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:15.667 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:15.665+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:15.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:15.758 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:15.758 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:15.768 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:15.765+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:15.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:15.851 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:15.868 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:15.865+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:15.870 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:15.879 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:15.879 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:15.879 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:15.879 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:15.879 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:15.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:15.891 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:15.912 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:15.934 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:15.958 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:15.969 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:15.965+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:15.996 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.169248858000003, 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:16:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.168633997000001, 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:16:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:16:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.176778841999997, 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.171890529000002, 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:16:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.166623909999998, 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.164349895999997, 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:16:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:16:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.023 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:16.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:16.069 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:16.065+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:16.171 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:16.169+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:16.175 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:16.176 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:16.176 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:16.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:16.272 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:16.269+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:16.303 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:16.303 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:16.303 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:16.303 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:16.303 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:16.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:16.373 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:16.373+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.684319428999999, 2024-09-21T15:16:16.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.472 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.550509058999999, 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:16.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:16.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.477 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.550449831999998, 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:16.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:16.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:16.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:16.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:16.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.436127003999999, 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:16.483 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:16.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:16.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:16.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.228559142999998, 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.228389202999999, 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.494 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.228360303000002, 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.228308828000003, 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:16.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:16.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.510 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.228130137999997, 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:16.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:16.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:16.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:16.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.228047330000003, 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:16.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:16.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:16.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:16.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.227889474000001, 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:16.521 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.526 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.227764763000003, 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:16.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.227734710999997, 2024-09-21T15:16:16.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.537 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.781526782, 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:16.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:16.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:16.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:16.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:16.543 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.781486172999998, 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:16.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:16.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.781434761, 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.549 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.554 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.781249797999997, 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:16.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:16.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.781220265999998, 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.781177569999997, 2024-09-21T15:16:16.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.780935898000003, 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.576 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:16.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:16.578 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:16.573+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:16.678 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:16.677+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:16.694 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:16.694 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:16.694 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:16.694 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:16.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:16.779 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:16.777+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:16.812 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:16.812 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:16.812 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:16.812 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:16.812 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:16.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:16.880 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:16.877+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:16.932 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:16.932 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:16.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:16.932 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.112350091000003, 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:16.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:16.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:16.937 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:16.942 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:16.989 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:16.989+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:17.060 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.060 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:17.060 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:17.060 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:17.082 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:17.077+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:17.173 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:17.174 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.182 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:17.181+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:17.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:17.283 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:17.281+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:17.292 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.293 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:17.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:17.293 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:16:17.293 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:17.384 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:17.381+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:17.455 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:17.455 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:17.455 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:17.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:16:17.513 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:17.513+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:17.569 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.569 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:17.570 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:17.570 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:17.570 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.578 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:17.585 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:17.581+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:17.686 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:17.685+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.868845559, 2024-09-21T15:16:17.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:17.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:17.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 52.859288974999998, 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:17.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:17.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:17.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:17.697 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:17.787 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:17.785+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:17.799 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.800 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:17.800 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:17.800 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.809 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:16:17.888 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:17.885+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:17.918 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:17.918 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:17.918 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:17.918 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:17.918 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:17.928 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:17.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:17.988 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:17.985+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:18.034 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:18.034 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:18.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.137362099999997, 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:18.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.115197922, 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.039 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:18.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.113751579999999, 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.043 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:18.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:18.047 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.112778024000001, 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:18.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:18.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.051 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.110351164000001, 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:18.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:18.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:18.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.109643349000002, 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.056 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:18.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.037125824, 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:18.060 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:18.064 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.036333118000002, 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:18.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.035166717000003, 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.070 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.033375855999999, 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.074 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:18.075 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.076 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.002759478000002, 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.077 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.079 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.000791268, 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:18.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:18.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.999038249000002, 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:18.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:18.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.997653608999997, 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:18.089 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:18.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:18.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.796805155999998, 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.097 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.796040222999999, 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:18.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.795029833999997, 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.102 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:18.106 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.794169427, 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:18.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.792839555999997, 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:18.110 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:18.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.791380136999997, 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:18.114 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:18.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:18.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:18.118 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:18.119 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:18.117+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:18.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:18.129 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:18.152 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:18.171 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:18.195 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:18.219 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:18.219 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:18.217+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:18.231 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:18.231 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:18.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:18.243 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:18.244 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:18.245 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:18.320 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:18.317+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:18.344 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:18.344 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:18.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:18.421 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:18.421+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:18.465 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:18.465 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:18.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:18.522 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:18.521+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:18.581 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:18.581 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:18.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:18.623 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:18.621+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:18.694 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:18.694 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:18.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:18.724 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:18.721+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:18.809 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:18.809 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:18.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:18.825 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:18.825+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:18.924 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:18.924 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:18.925 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:18.925+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:18.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:19.026 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:19.025+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:19.038 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.038 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:19.127 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:19.125+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:19.156 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.156 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.166 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:19.228 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:19.225+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:19.284 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.284 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.295 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:19.329 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:19.325+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:19.402 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.402 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.411 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:19.457 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:19.457+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:19.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:19.509 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:19.509 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:19.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-09-21T15:16:19.516 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.516 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.526 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:19.554 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:19.553+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:19.631 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:19.629+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:19.634 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.634 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.645 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:19.732 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:19.729+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:19.748 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.748 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.758 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:19.833 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:19.829+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:19.838 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-09-21T15:16:19.837+0000 7f6b15ef9700 -1 mon.a@0(leader).osd e42 definitely_dead 0 2024-09-21T15:16:19.860 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.860 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:19.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:16:19.897 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.2. 2024-09-21T15:16:19.934 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:19.933+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:19.974 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:19.974 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:19.984 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:20.034 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:20.033+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:20.088 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:20.088 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:20.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:20.135 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:20.133+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:20.200 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:20.200 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:20.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:20.254 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:20.253+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:20.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:20.320 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:20.320 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:20.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:20.337 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:20.333+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:20.392 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:20.411 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:20.430 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:20.434 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:20.434 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:20.437 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:20.433+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:20.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:20.453 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:20.493 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:20.520 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:20.538 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:20.537+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:20.552 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:20.552 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.725478015999997, 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:20.553 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:16:20.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:16:20.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.724863155000001, 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:20.557 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:16:20.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:16:20.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.733007999999998, 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:20.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:20.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:20.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:20.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.728119687000003, 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:16:20.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:16:20.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.722853067999999, 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:20.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:20.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.720579053999998, 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.576 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.580 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:20.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:20.639 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:20.637+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:20.736 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:20.736 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:20.736 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:20.736 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:20.740 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:20.737+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:20.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:20.842 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:20.841+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:20.850 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:20.850 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:20.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:20.904 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:16:20.941 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:20.941+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:20.971 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:20.971 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.188096162000001, 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:20.972 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:20.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:20.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.054285792000002, 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:20.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:20.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.054226565, 2024-09-21T15:16:20.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:20.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:20.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:20.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:20.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:20.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:20.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:20.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:20.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:20.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:20.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:20.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:20.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:20.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:20.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:20.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:20.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:20.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.994 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:20.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:20.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.994 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:20.995 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:20.995 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.939903737000002, 2024-09-21T15:16:20.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:20.995 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:20.995 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:20.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:20.996 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:20.996 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:20.996 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:20.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.996 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:20.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:20.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:20.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:20.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:20.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:20.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:20.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:20.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:20.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:20.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:21.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:21.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:21.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:21.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:21.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:21.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:21.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:21.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:21.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:21.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:21.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:21.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:21.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:21.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:21.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:21.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:21.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:21.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.732335876, 2024-09-21T15:16:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.732165936000001, 2024-09-21T15:16:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.732137035999997, 2024-09-21T15:16:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.732085560999998, 2024-09-21T15:16:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.731906871, 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.731824062999998, 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.731666207000004, 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.731541495999998, 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.731511443999999, 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.285303515000003, 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.285262906, 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.285211494000002, 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.285026531, 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.284996999000001, 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.284954302999999, 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.284712630999998, 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:21.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.124 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:21.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:21.126 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:21.125+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:21.226 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:21.225+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:21.245 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:21.245 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:21.245 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:21.246 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:21.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:21.327 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:21.325+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:21.409 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:21.410 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:21.410 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:21.410 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:21.410 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:21.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:21.427 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:21.425+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:21.524 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.704353132999998, 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:21.529 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:21.530 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:21.529+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:21.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:21.631 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:21.629+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:21.650 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:21.650 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:21.651 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:21.651 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:21.661 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:21.732 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:21.729+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:21.768 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:21.768 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:21.778 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:21.834 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:21.829+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:21.882 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:21.882 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:21.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:21.882 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:16:21.882 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:21.892 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:21.935 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:21.933+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:21.996 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:21.996 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:21.996 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:21.996 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:22.006 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:22.036 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:22.033+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:22.107 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:22.107 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:22.118 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:22.137 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:22.133+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:22.222 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.403309659999998, 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 57.393753076000003, 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.72053165200000002, 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.236 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:22.237 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:22.238 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:22.237+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:22.339 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:22.337+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:22.340 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:22.341 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:22.341 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:22.341 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:22.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:22.440 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:22.437+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:22.454 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:22.454 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:22.454 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:22.454 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:22.454 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:22.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:22.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.673250076000002, 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.570 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:22.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:22.574 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.651085898000005, 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:22.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:22.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:22.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:22.580 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.649639555999997, 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.648666000000006, 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.646239140000006, 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.592 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.645531324999993, 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:22.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:22.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.573013799999998, 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:22.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:22.600 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.572221094, 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:22.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:22.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:22.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.571054692999994, 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.569263832000004, 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:22.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:22.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:22.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.538647453999999, 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.536679243999998, 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.620 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.534926225000007, 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.533541584999995, 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:22.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.628 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.332693132000003, 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:22.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:22.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:22.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.331928199000004, 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:22.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:22.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:22.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.330917810000003, 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:22.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.640 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.330057402999998, 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:22.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:22.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.328727532000002, 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.327268113000002, 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:22.649 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:22.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:22.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:22.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:22.653 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:22.654 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:22.581+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:22.655 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:22.653+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:22.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:22.688 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:22.708 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:22.733 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:22.754 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:22.755 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:22.753+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:22.761 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:22.761 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:22.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:22.783 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:22.811 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:22.857 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:22.853+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:22.892 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:22.892 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:22.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:22.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:16:22.957 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:22.957+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:23.008 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.008 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:23.057 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:23.053+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:23.116 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.117 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:23.159 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:23.157+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:23.238 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.238 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:23.259 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:23.257+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:23.361 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:23.361+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:23.401 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.401 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:23.461 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:23.457+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:23.520 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.520 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:23.562 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:23.561+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:23.639 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.639 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:23.662 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:23.661+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:23.757 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.757 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.763 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:23.761+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:23.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:23.865 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:23.865+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:23.884 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.884 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:23.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:16:23.966 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:23.965+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:23.997 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:23.997 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.007 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:24.067 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:24.065+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:24.115 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:24.115 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.123 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:24.168 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:24.165+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:24.235 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:24.235 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:24.293 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:24.293+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:24.370 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:24.369+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:24.405 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:24.405 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:24.470 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:24.469+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:24.522 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:24.522 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.531 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:24.572 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:24.573+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:24.672 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:24.669+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:24.680 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:24.680 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.689 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:24.773 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:24.769+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:24.794 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:24.794 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.804 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:24.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:24.875 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:24.873+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:24.905 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:24.905 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:24.911 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:24.912 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:24.912 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T15:16:24.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:16:24.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:24.975 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:24.973+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:24.978 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:24.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:25.023 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:25.023 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:25.023 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.024 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:25.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:25.048 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:25.072 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:25.076 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:25.073+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:25.094 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:25.096 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:25.096 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:25.096 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:25.096 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:25.096 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:25.096 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:25.148 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.149 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:25.149 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:25.149 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:25.149 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:25.177 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:25.177+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 45, 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:16:23.908812+0000", 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:16:19.883269+0000", 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5806080, 2024-09-21T15:16:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 2, 2024-09-21T15:16:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "jewel", 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:16:25.207 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:16:25.208 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:16:25.209 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:16:25.210 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:16:25.211 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:16:25.212 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:16:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:16:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:16:25.215 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 20, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:16:25.216 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:16:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.2699999809265137, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.2699999809265137, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:25.219 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.220 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.223 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:25.224 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 43, 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.225 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:16:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 0, 2024-09-21T15:16:25.229 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:16:25.230 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:16:25.231 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:16:25.232 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:25.234 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:25.235 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:25.236 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:16:25.239 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.241 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.242 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.243 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:16:25.244 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:25.245 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 43, 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:16:25.246 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.249 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:25.250 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:16:25.251 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 65536, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:25.252 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [], 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [], 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:16:25.254 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:16:25.255 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:16:25.256 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:16:25.256 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:16:25.256 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:16:25.256 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:16:25.256 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:16:25.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.256 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.257 INFO:tasks.thrashosds.thrasher:j is {'epoch': 45, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:16:23.908812+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:16:19.883269+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 43, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 43, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:16:25.257 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T15:16:25.278 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:25.277+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:25.308 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.480875949999998, 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.480261089000003, 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:16:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.488405933999999, 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:16:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.483517620999997, 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.478251002, 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.475976987999999, 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:16:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:16:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:16:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.336 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:25.378 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:25.377+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:25.448 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:25.448 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:25.480 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:25.477+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:25.580 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:25.577+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:25.610 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:25.610 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:25.611 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:25.611 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:25.681 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:25.681+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:25.732 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.732 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:25.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.95058607, 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.733 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:25.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:25.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.816775699999994, 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.739 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:25.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:25.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:25.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.816716473, 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:25.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:25.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:25.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:25.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.702393645000001, 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:25.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:25.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494825784, 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.756 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:25.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:25.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:25.761 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494655843999993, 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:25.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494626944000004, 2024-09-21T15:16:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:25.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.772 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494575468999997, 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.773 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:25.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:25.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.778 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494396778999999, 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:25.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:25.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:25.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:25.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:25.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494313970999997, 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:25.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.790 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494156114999996, 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:25.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:25.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:25.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:25.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494031403999998, 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.796 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:25.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:25.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:25.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.494001351999998, 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:25.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:25.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:25.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:25.806 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.047793423000002, 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.807 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.047752814000006, 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.047701402000001, 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.047516439000006, 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.047486907000007, 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.047444210999998, 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.047202538999997, 2024-09-21T15:16:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:25.845 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:25.847 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:25.845+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:25.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:16:25.946 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:25.945+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:25.974 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:25.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:26.047 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:26.045+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:26.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:26.148 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:26.145+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:26.249 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:26.249+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:26.269 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.449190643000001, 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:26.274 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:26.279 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:26.350 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:26.349+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:26.406 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:26.451 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:26.453+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:26.510 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:26.510 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:26.510 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:26.510 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:26.510 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:26.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:26.552 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:26.549+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:26.629 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:26.630 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:26.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:26.630 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:16:26.630 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:26.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:26.653 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:26.653+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:26.748 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:26.748 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:26.748 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:26.748 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:26.753 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:26.753+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:26.757 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:26.854 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:26.853+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:26.916 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:26.916 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:26.916 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:26.916 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:26.916 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:26.925 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:26.946 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:16:26.955 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:26.953+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.211484734000003, 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:27.031 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:27.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:27.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.035 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 62.201928150000001, 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:27.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:27.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:27.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5287067260000002, 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:27.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:27.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:27.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:27.043 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:27.044 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:27.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:27.089 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:27.089+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:27.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:27.148 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:27.149 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:27.149 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:27.149 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:27.157 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:27.157+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:27.159 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:27.258 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:27.253+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:27.268 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:27.268 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:27.268 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:27.268 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:27.268 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:27.268 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:27.278 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:27.291 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:27.310 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:27.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:27.355 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:27.358 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:27.357+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:27.377 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:27.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:27.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:27.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:27.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:27.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:27.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:27.388 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:27.388 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:27.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:27.459 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:27.457+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:27.514 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:27.514 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:27.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:27.561 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:27.557+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:27.637 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:27.637 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:27.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:27.661 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:27.661+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:27.763 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:27.761+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:27.800 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:27.800 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:27.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:27.863 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:27.861+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:27.917 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:27.917 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:27.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:27.965 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:27.965+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.036 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:28.065 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:28.061+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:28.151 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.151 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:28.202 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:28.201+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:28.266 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:28.265+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:28.312 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.312 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:28.367 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:28.365+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:28.428 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.429 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:28.468 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:28.465+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:28.542 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.543 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:28.597 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:28.597+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:28.670 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:28.669+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:28.671 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.671 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:28.771 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:28.769+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:28.793 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.794 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.803 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:28.872 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:28.869+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:28.912 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:28.913 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:28.922 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:28.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:16:29.010 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:29.009+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:29.073 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:29.074 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:29.074 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:29.073+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:29.083 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:29.175 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:29.173+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:29.191 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:29.191 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:29.201 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:29.275 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:29.273+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:29.312 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:29.312 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:29.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:29.376 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:29.377+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:29.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:29.424 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:29.424 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:29.434 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:29.478 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:29.473+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:29.534 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:29.534 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:29.540 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:29.543 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:29.562 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:29.583 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:29.589 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:29.589+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:29.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:29.629 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:29.652 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:29.657 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.760896549999998, 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.738732372000001, 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.737286030000007, 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.736312474000002, 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.733885614000002, 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.678 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.733177799000003, 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:29.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.682 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.660660273999994, 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.686 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.659867567999996, 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:29.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:29.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.658701167000004, 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:29.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:29.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.656910306, 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:29.695 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:29.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:29.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:29.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.626293927999996, 2024-09-21T15:16:29.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.703 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.624325717999994, 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:29.707 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.622572699000003, 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:29.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:29.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.621188059000005, 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:29.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.420339605999999, 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:29.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:29.720 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.419574673, 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.418564283999999, 2024-09-21T15:16:29.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.728 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.417703876999994, 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:29.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.416374005999998, 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:29.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:29.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:29.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:29.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:29.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:29.737 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.737 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.738 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.738 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:29.738 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:29.738 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.414914586999998, 2024-09-21T15:16:29.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:29.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:29.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:29.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:29.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:29.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:29.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:29.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:29.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:29.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:29.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:29.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:29.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:29.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:29.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:29.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:29.745 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.745 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.745 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.745 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:29.745 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:29.745 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:29.747 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:29.745+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:29.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:29.847 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:29.845+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:29.918 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:29.918 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:29.918 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:29.918 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:29.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:29.948 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:29.945+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:16:30.033 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:30.033 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:30.033 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:30.033 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:30.033 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:30.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:30.049 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:30.049+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:30.147 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.320016891999998, 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:16:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:16:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.319402030999996, 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:16:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.327546876, 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.322658563000005, 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.317391943999993, 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.31511793, 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.175 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:30.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:30.177 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:30.177+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:30.258 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:30.259 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:30.259 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 3, 4, 5] 2024-09-21T15:16:30.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-09-21T15:16:30.278 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:30.273+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:30.288 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:30.289 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:30.289 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:30.289 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:30.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:30.379 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:30.377+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:30.463 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:30.463 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:30.463 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:30.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:30.518 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:30.517+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:30.580 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.796963891999994, 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.663153522000002, 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.663094294999993, 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.548771466999995, 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.341203605999993, 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.341033666000001, 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.341004765999998, 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.340953291000005, 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.340774601000007, 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.340691793000005, 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.340533937000004, 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:30.645 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.340409226000006, 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:30.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:30.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:30.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.340379174000006, 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:30.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:30.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.894171244999995, 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:30.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:30.661 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:30.577+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.663 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.894130636, 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:30.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:30.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.894079223999995, 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.669 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:30.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:30.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:30.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.893894261, 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.675 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:30.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.893864729000001, 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:30.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.893822033000006, 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:30.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:30.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:30.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.893580361000005, 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:30.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:30.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:30.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:30.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:30.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:30.697 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:30.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:30.697 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-09-21T15:16:30.589+0000 7f6b15ef9700 -1 mon.a@0(leader).osd e47 definitely_dead 0 2024-09-21T15:16:30.699 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:30.697+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:30.799 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:30.797+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:30.815 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:30.815 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:30.815 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:30.815 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:30.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:30.839 INFO:tasks.workunit.client.0.smithi146.stderr:Note: switching to 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'. 2024-09-21T15:16:30.839 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-21T15:16:30.839 INFO:tasks.workunit.client.0.smithi146.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr: git switch -c 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr:Or undo this operation with: 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr: git switch - 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-21T15:16:30.840 INFO:tasks.workunit.client.0.smithi146.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T15:16:30.841 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-21T15:16:30.841 INFO:tasks.workunit.client.0.smithi146.stderr:HEAD is now at a477b1894d5 Merge pull request #59761 from rhcs-dashboard/reef-hosts-selection-show-only-10-entries-fix 2024-09-21T15:16:30.845 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T15:16:30.900 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:30.897+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:30.930 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:30.930 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:30.930 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:30.930 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:30.930 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:30.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:30.982 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-09-21T15:16:31.001 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:31.001+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:31.068 INFO:tasks.workunit.client.0.smithi146.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T15:16:31.071 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T15:16:31.071 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T15:16:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.260303550000003, 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:31.085 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:31.090 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:31.102 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:31.101+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:31.203 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:31.201+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:31.210 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:31.210 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:31.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:31.304 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:31.301+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:31.324 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:31.324 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:31.324 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:31.324 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:31.324 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:31.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:31.404 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:31.401+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:31.486 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:31.486 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:31.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:31.486 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [] 2024-09-21T15:16:31.486 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:31.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:31.505 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:31.505+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:31.604 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:31.605 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:31.606 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:31.605+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:31.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:31.707 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:31.705+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:31.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:31.764 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:31.765 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:31.765 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:31.765 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:31.765 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:31.774 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:31.808 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:31.805+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:31.870 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.059251837000005, 2024-09-21T15:16:31.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:31.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:31.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:31.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.049695252999996, 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:31.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:31.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.376473829, 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:31.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:31.891 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:31.892 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:31.902 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:31.909 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:31.905+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:31.924 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:31.946 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:31.948 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T15:16:31.970 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:31.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:16:32.001 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:32.002 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:32.002 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:32.002 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:32.002 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:32.009 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:32.005+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:32.012 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:32.014 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T15:16:32.016 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T15:16:32.016 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T15:16:32.026 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:32.071 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T15:16:32.076 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-21T15:16:32.076 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T15:16:32.082 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-09-21T15:16:32.084 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-09-21T15:16:32.084 DEBUG:teuthology.orchestra.run.smithi146:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a477b1894d5c5a613f23260ca0e6b96d79652ba7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-21T15:16:32.110 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:32.109+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:32.112 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:32.112 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:32.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:32.140 INFO:tasks.workunit.client.0.smithi146.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-21T15:16:32.142 INFO:tasks.workunit.client.0.smithi146.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-09-21T15:16:32.142 INFO:tasks.workunit.client.0.smithi146.stderr:++ TIMEOUT=300 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ MAX_TIMEOUT=15 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ PG_NUM=4 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ TMPDIR=/tmp 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ type xmlstarlet 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ XMLSTARLET=xmlstarlet 2024-09-21T15:16:32.143 INFO:tasks.workunit.client.0.smithi146.stderr:+++ uname 2024-09-21T15:16:32.144 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' Linux = FreeBSD ']' 2024-09-21T15:16:32.144 INFO:tasks.workunit.client.0.smithi146.stderr:++ SED=sed 2024-09-21T15:16:32.144 INFO:tasks.workunit.client.0.smithi146.stderr:++ AWK=awk 2024-09-21T15:16:32.144 INFO:tasks.workunit.client.0.smithi146.stderr:+++ stty -a 2024-09-21T15:16:32.144 INFO:tasks.workunit.client.0.smithi146.stderr:+++ head -1 2024-09-21T15:16:32.145 INFO:tasks.workunit.client.0.smithi146.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-21T15:16:32.156 INFO:tasks.workunit.client.0.smithi146.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-21T15:16:32.157 INFO:tasks.workunit.client.0.smithi146.stderr:++ termwidth= 2024-09-21T15:16:32.157 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-21T15:16:32.157 INFO:tasks.workunit.client.0.smithi146.stderr:++ DIFFCOLOPTS='-y ' 2024-09-21T15:16:32.157 INFO:tasks.workunit.client.0.smithi146.stderr:++ KERNCORE=kernel.core_pattern 2024-09-21T15:16:32.157 INFO:tasks.workunit.client.0.smithi146.stderr:++ EXTRA_OPTS= 2024-09-21T15:16:32.168 INFO:tasks.workunit.client.0.smithi146.stderr:++ test '' = TESTS 2024-09-21T15:16:32.169 INFO:tasks.workunit.client.0.smithi146.stderr:+ POOL=rbd 2024-09-21T15:16:32.169 INFO:tasks.workunit.client.0.smithi146.stderr:+ ANOTHER_POOL=new_default_pool17425 2024-09-21T15:16:32.169 INFO:tasks.workunit.client.0.smithi146.stderr:+ NS=ns 2024-09-21T15:16:32.169 INFO:tasks.workunit.client.0.smithi146.stderr:+ IMAGE=testrbdnbd17425 2024-09-21T15:16:32.169 INFO:tasks.workunit.client.0.smithi146.stderr:+ SIZE=64 2024-09-21T15:16:32.169 INFO:tasks.workunit.client.0.smithi146.stderr:+ DATA= 2024-09-21T15:16:32.169 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:16:32.170 INFO:tasks.workunit.client.0.smithi146.stderr:+ setup 2024-09-21T15:16:32.170 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns x 2024-09-21T15:16:32.170 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -e CMakeCache.txt ']' 2024-09-21T15:16:32.170 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo echo test sudo 2024-09-21T15:16:32.170 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:32.171 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:32.172 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:32.172 INFO:tasks.workunit.client.0.smithi146.stderr:++ which echo 2024-09-21T15:16:32.175 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/echo 2024-09-21T15:16:32.175 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:32.175 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-09-21T15:16:32.180 INFO:tasks.workunit.client.0.smithi146.stdout:test sudo 2024-09-21T15:16:32.180 INFO:tasks.workunit.client.0.smithi146.stderr:+ trap cleanup INT TERM EXIT 2024-09-21T15:16:32.181 INFO:tasks.workunit.client.0.smithi146.stderr:++ mktemp -d 2024-09-21T15:16:32.182 INFO:tasks.workunit.client.0.smithi146.stderr:+ TEMPDIR=/tmp/tmp.flUinkcHzA 2024-09-21T15:16:32.182 INFO:tasks.workunit.client.0.smithi146.stderr:+ DATA=/tmp/tmp.flUinkcHzA/data 2024-09-21T15:16:32.182 INFO:tasks.workunit.client.0.smithi146.stderr:+ dd if=/dev/urandom of=/tmp/tmp.flUinkcHzA/data bs=1M count=64 2024-09-21T15:16:32.211 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:32.209+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:32.224 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:32.224 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:32.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:32.313 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:32.309+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:32.341 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:32.341 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:32.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:32.385 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:32.385 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:32.385 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.201719 s, 333 MB/s 2024-09-21T15:16:32.386 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd namespace create rbd/ns 2024-09-21T15:16:32.414 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:32.413+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:32.459 INFO:tasks.workunit.client.0.smithi146.stderr:+ for ns in '' ${NS} 2024-09-21T15:16:32.459 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.flUinkcHzA/data testrbdnbd17425 2024-09-21T15:16:32.509 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:32.509 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:32.515 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:32.513+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:32.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:32.615 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:32.613+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:32.631 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:32.631 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:32.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:32.716 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:32.717+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:32.796 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:32.796 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:32.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:32.817 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:32.817+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:32.918 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:32.918 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:32.918 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:32.917+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:32.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:33.010 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:16:33.018 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:33.017+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:33.047 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.047 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:33.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:33.119 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:33.117+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:33.215 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.215 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:33.221 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:33.217+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:33.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:33.322 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:33.321+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:33.353 INFO:tasks.workunit.client.0.smithi146.stderr:+ for ns in '' ${NS} 2024-09-21T15:16:33.353 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.flUinkcHzA/data testrbdnbd17425 2024-09-21T15:16:33.373 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.374 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:33.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:33.423 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:33.421+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:33.496 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.496 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:33.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:33.553 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:33.553+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:33.614 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.614 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:33.624 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:33.625 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:33.625+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:33.726 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.726 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:33.726 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:33.725+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:33.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:33.827 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:33.825+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:33.844 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.844 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:33.853 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:33.927 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:33.925+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:33.991 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:33.991 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:34.001 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:34.028 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:34.025+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:34.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:34.099 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:34.100 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:34.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:34.128 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:34.129+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:34.180 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:34.199 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:34.212 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:34.212 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:34.219 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:34.221 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:34.229 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:34.225+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:34.240 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:34.262 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:34.285 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:34.287 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph osd pool create new_default_pool17425 8 2024-09-21T15:16:34.288 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:34.288 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:34.288 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:34.288 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:34.288 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:34.288 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:34.320 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:34.320 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:34.320 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:34.320 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:34.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:34.330 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:34.329+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:34.431 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:34.429+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:34.432 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:34.432 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:34.441 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:34.532 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:34.529+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:34.547 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:34.547 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:34.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:34.547 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:34.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.547 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.650789802000006, 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:34.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.551 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.628625623999994, 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:34.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:34.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.627179282, 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:34.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.626205725999995, 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.560 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:34.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:34.564 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.623778865999995, 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:34.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:34.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.568 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.623071050999997, 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:34.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.550553526000002, 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:34.576 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.549760820000003, 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:34.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:34.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:34.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:34.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.548594418999997, 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:34.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:34.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.546803557999993, 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.585 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:34.589 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.516187180000003, 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:34.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:34.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.593 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.514218970000002, 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:34.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.512465950999996, 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:34.598 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.602 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.511081310999998, 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:34.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:34.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.310232858000006, 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:34.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:34.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:34.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.309467925000007, 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.308457536000006, 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:34.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.307597129000001, 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.619 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:34.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:34.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.306267258000005, 2024-09-21T15:16:34.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:34.627 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.304807839000006, 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:34.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:34.631 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:34.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:34.633 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:34.629+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:34.734 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:34.733+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:34.749 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:34.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:34.749 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:34.749 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:34.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:34.835 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:34.833+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:34.869 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:34.869 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:34.869 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:34.870 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:34.870 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:34.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:34.936 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:34.933+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.162519372000006, 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041796200000000002, 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829101+0000", 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829100+0000", 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829109+0000", 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2739999999999996e-06 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829118+0000", 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7770000000000008e-06 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829129+0000", 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1388e-05 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829379+0000", 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025028700000000002 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829519+0000", 2024-09-21T15:16:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000139629 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18)", 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.161904511000003, 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023011400000000001, 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829716+0000", 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829715+0000", 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829722+0000", 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8190000000000003e-06 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829747+0000", 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5477999999999999e-05 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829758+0000", 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1222e-05 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829806+0000", 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8180999999999998e-05 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.829946+0000", 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013951999999999999 2024-09-21T15:16:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.170049356000007, 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:5 2.2 e19/17 ondisk, result = 0)", 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.165161042999998, 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019988300000000001, 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826459+0000", 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826458+0000", 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826468+0000", 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0128e-05 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826489+0000", 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.156e-05 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826504+0000", 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4902999999999999e-05 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826558+0000", 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3300000000000001e-05 2024-09-21T15:16:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826566+0000", 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3319999999999992e-06 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826659+0000", 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3315999999999999e-05 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.159894424000001, 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4208.0:7 2.2 e20/17 ondisk, result = 0)", 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.157620410000007, 2024-09-21T15:16:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018459499999999999, 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834000+0000", 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833999+0000", 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834011+0000", 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.259e-05 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834025+0000", 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3550000000000001e-05 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834034+0000", 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6999999999999997e-06 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834081+0000", 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7793000000000001e-05 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834090+0000", 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4319999999999999e-06 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834185+0000", 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4605999999999995e-05 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4986.0:61 2.5 e50/48 2:a72945ab:ns::rbd_object_map.137aae78387e:head v 50'25, mlcod=50'23)", 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.271029+0000", 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.72059155900000005, 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012974379999999999, 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271029+0000", 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271029+0000", 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271027+0000", 2024-09-21T15:16:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271048+0000", 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0746000000000001e-05 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271080+0000", 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2938999999999998e-05 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271093+0000", 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2833000000000001e-05 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271202+0000", 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108365 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271244+0000", 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2738999999999999e-05 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.272251+0000", 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010064080000000001 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.272326+0000", 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5403999999999998e-05 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4986.0:62 2.6 2:6e955b78:ns::rbd_header.137aae78387e:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T15:16:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.273065+0000", 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.71855520299999998, 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020092360000000002, 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4986", 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1464530257", 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273065+0000", 2024-09-21T15:16:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273065+0000", 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273062+0000", 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273079+0000", 2024-09-21T15:16:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273112+0000", 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2576000000000001e-05 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273128+0000", 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5702999999999999e-05 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273228+0000", 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100721 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273380+0000", 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151197 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273650+0000", 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026987700000000002 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274923+0000", 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012735820000000001 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274936+0000", 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3040000000000001e-05 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275016+0000", 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0224999999999998e-05 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275074+0000", 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7969e-05 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4986.0:62 2.6 e50/48 ondisk, result = 0)", 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.274774+0000", 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.71684599699999996, 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000311436, 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:35.029 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274774+0000", 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274774+0000", 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274773+0000", 2024-09-21T15:16:35.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274786+0000", 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853e-05 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274802+0000", 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5688e-05 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274812+0000", 2024-09-21T15:16:35.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0521e-05 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274904+0000", 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1267000000000006e-05 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274915+0000", 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.186e-05 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275086+0000", 2024-09-21T15:16:35.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000170316 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4986.0:63 2.6 2:6e955b78:ns::rbd_header.137aae78387e:head [notify cookie 139885407124624] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.275510+0000", 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.71611076299999998, 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00071835099999999999, 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4986", 2024-09-21T15:16:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1464530257", 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 63 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275510+0000", 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275510+0000", 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275508+0000", 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275517+0000", 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2139999999999996e-06 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275537+0000", 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0360000000000002e-05 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275548+0000", 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0755e-05 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275651+0000", 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010283799999999999 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275882+0000", 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000231888 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276228+0000", 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034548599999999999 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4986.0:64 2.6 2:6e955b78:ns::rbd_header.137aae78387e:head [watch unwatch cookie 139885407133504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.275578+0000", 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.71604263700000004, 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026242650000000002, 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.037 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4986", 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1464530257", 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 64 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275578+0000", 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275578+0000", 2024-09-21T15:16:35.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275577+0000", 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275586+0000", 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6320000000000003e-06 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275598+0000", 2024-09-21T15:16:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1372000000000001e-05 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275608+0000", 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0185e-05 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276265+0000", 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065655399999999995 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276412+0000", 2024-09-21T15:16:35.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014707699999999999 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276640+0000", 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227879 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277727+0000", 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001087066 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277802+0000", 2024-09-21T15:16:35.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4987999999999994e-05 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277941+0000", 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013944399999999999 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278202+0000", 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260931 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4986.0:65 2.6 2:6e955b78:ns::rbd_header.137aae78387e:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.276487+0000", 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.71513292900000003, 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020420500000000001, 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4986", 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1464530257", 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 65 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276487+0000", 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276487+0000", 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276486+0000", 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276494+0000", 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0080000000000002e-06 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276520+0000", 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5616e-05 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276530+0000", 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277062+0000", 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053193000000000001 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277335+0000", 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000272417 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278281+0000", 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000946702 2024-09-21T15:16:35.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278401+0000", 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119368 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278529+0000", 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012877400000000001 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.047 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.4986.0:64 2.6 e50/48 ondisk, result = 0)", 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.277655+0000", 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.71396539400000003, 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00057344399999999999, 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277655+0000", 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277655+0000", 2024-09-21T15:16:35.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277654+0000", 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277664+0000", 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8509999999999994e-06 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277693+0000", 2024-09-21T15:16:35.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.906e-05 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277702+0000", 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9069999999999996e-06 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277770+0000", 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7685e-05 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277787+0000", 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7614000000000001e-05 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278228+0000", 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000441156 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.051 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:35.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:16:35.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:35.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:35.049+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:35.127 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'new_default_pool17425' already exists 2024-09-21T15:16:35.137 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd pool init new_default_pool17425 2024-09-21T15:16:35.154 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:35.153+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:35.167 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:35.167 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:35.167 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:35.167 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:35.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:35.255 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:35.253+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:35.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:35.356 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:35.353+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.610049711000002, 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.476239340999996, 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.476180114000002, 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.361857286000003, 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.154289425000002, 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.154119484999995, 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.154090585000006, 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.154039109999999, 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.153860420000001, 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.153777611999999, 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.153619755999998, 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.153495045, 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.153464993, 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.707257064000004, 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.707216454999994, 2024-09-21T15:16:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.707165043000003, 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.706980079999994, 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.706950547999995, 2024-09-21T15:16:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.706907852000001, 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.706666179999999, 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.503 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:35.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:35.505 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:35.501+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:35.605 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:35.600+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:35.660 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:35.660 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:35.660 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:35.660 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:35.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:35.706 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:35.705+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:35.777 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:35.777 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:35.777 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:35.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:35.807 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:35.804+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:35.896 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.076626942000004, 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:35.897 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:35.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:35.901 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:35.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:35.908 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:35.905+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:35.994 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:35.994 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:35.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:16:36.008 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:36.004+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:36.028 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:16:36.030 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:36.030 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:36.030 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:36.030 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:36.043 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:36.109 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:36.104+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:36.146 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:36.146 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:36.146 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:36.146 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:36.146 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:36.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:36.210 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:36.204+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:36.266 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:36.266 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:36.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:36.266 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:36.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.266 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4986.0:62 2.6 e50/48 2:6e955b78:ns::rbd_header.137aae78387e:head v 50'16, mlcod=50'14)", 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:34.272128+0000", 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.994633825, 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086998900000000005, 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272128+0000", 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:36.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272128+0000", 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272127+0000", 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272139+0000", 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1398e-05 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272158+0000", 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9040000000000001e-05 2024-09-21T15:16:36.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272167+0000", 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7219999999999994e-06 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272223+0000", 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6066000000000002e-05 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272250+0000", 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6925000000000001e-05 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272971+0000", 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00072107499999999995 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.272998+0000", 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6923e-05 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4986.0:64 2.6 e50/48 2:6e955b78:ns::rbd_header.137aae78387e:head v 50'17, mlcod=50'15)", 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:34.275115+0000", 2024-09-21T15:16:36.270 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.991647655, 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00077479800000000004, 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275115+0000", 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275115+0000", 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275113+0000", 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275123+0000", 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7350000000000008e-06 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275144+0000", 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1319999999999999e-05 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:36.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275154+0000", 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2320000000000007e-06 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275189+0000", 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5349000000000003e-05 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275202+0000", 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3089e-05 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275861+0000", 2024-09-21T15:16:36.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00065870900000000005 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:34.275889+0000", 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8470999999999999e-05 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:36.274 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:36.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:36.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:36.314 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:36.312+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:36.317 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:16:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-21T15:16:36.022325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":126620,"kb_used_data":2148,"kb_used_omap":6,"kb_used_meta":124409,"kb_avail":377360740,"statfs":{"total":386547056640,"available":386417397760,"internally_reserved":0,"allocated":2199552,"data_stored":1924161,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":6373,"internal_metadata":127395611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.087926"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":192,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-21T15:16:29.965216+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:16:29.965216+0000","last_peered":"2024-09-21T15:16:29.965216+0000","last_clean":"2024-09-21T15:16:29.965216+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:16:29.965216+0000","last_undegraded":"2024-09-21T15:16:29.965216+0000","last_fullsized":"2024-09-21T15:16:29.965216+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":58,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-21T15:16:30.976940+0000","last_change":"2024-09-21T15:16:30.977084+0000","last_active":"2024-09-21T15:16:30.976940+0000","last_peered":"2024-09-21T15:16:29.965270+0000","last_clean":"2024-09-21T15:16:29.965270+0000","last_became_active":"2024-09-21T15:16:21.501873+0000","last_became_peered":"2024-09-21T15:16:21.501873+0000","last_unstale":"2024-09-21T15:16:30.976940+0000","last_undegraded":"2024-09-21T15:16:30.976940+0000","last_fullsized":"2024-09-21T15:16:30.976940+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:05:45.622685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-21T15:16:30.975950+0000","last_change":"2024-09-21T15:16:30.976116+0000","last_active":"2024-09-21T15:16:30.975950+0000","last_peered":"2024-09-21T15:16:27.037289+0000","last_clean":"2024-09-21T15:16:27.037289+0000","last_became_active":"2024-09-21T15:16:21.499763+0000","last_became_peered":"2024-09-21T15:16:21.499763+0000","last_unstale":"2024-09-21T15:16:30.975950+0000","last_undegraded":"2024-09-21T15:16:30.975950+0000","last_fullsized":"2024-09-21T15:16:30.975950+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:14.573674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T15:16:27.037362+0000","last_change":"2024-09-21T15:15:34.263162+0000","last_active":"2024-09-21T15:16:27.037362+0000","last_peered":"2024-09-21T15:16:27.037362+0000","last_clean":"2024-09-21T15:16:27.037362+0000","last_became_active":"2024-09-21T15:15:34.262327+0000","last_became_peered":"2024-09-21T15:15:34.262327+0000","last_unstale":"2024-09-21T15:16:27.037362+0000","last_undegraded":"2024-09-21T15:16:27.037362+0000","last_fullsized":"2024-09-21T15:16:27.037362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:15.380455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":45,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-21T15:16:30.978787+0000","last_change":"2024-09-21T15:16:30.978942+0000","last_active":"2024-09-21T15:16:30.978787+0000","last_peered":"2024-09-21T15:16:29.967495+0000","last_clean":"2024-09-21T15:16:29.967495+0000","last_became_active":"2024-09-21T15:15:22.829001+0000","last_became_peered":"2024-09-21T15:15:22.829001+0000","last_unstale":"2024-09-21T15:16:30.978787+0000","last_undegraded":"2024-09-21T15:16:30.978787+0000","last_fullsized":"2024-09-21T15:16:30.978787+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:10.307303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T15:16:27.037341+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:16:27.037341+0000","last_peered":"2024-09-21T15:16:27.037341+0000","last_clean":"2024-09-21T15:16:27.037341+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:16:27.037341+0000","last_undegraded":"2024-09-21T15:16:27.037341+0000","last_fullsized":"2024-09-21T15:16:27.037341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-21T15:16:30.977443+0000","last_change":"2024-09-21T15:16:30.977710+0000","last_active":"2024-09-21T15:16:30.977443+0000","last_peered":"2024-09-21T15:16:29.965182+0000","last_clean":"2024-09-21T15:16:29.965182+0000","last_became_active":"2024-09-21T15:15:22.823463+0000","last_became_peered":"2024-09-21T15:15:22.823463+0000","last_unstale":"2024-09-21T15:16:30.977443+0000","last_undegraded":"2024-09-21T15:16:30.977443+0000","last_fullsized":"2024-09-21T15:16:30.977443+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:10.962624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-21T15:16:30.978523+0000","last_change":"2024-09-21T15:16:30.978523+0000","last_active":"2024-09-21T15:15:21.793256+0000","last_peered":"2024-09-21T15:15:21.793256+0000","last_clean":"2024-09-21T15:15:21.793256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:30.978523+0000","last_undegraded":"2024-09-21T15:16:30.978523+0000","last_fullsized":"2024-09-21T15:16:30.978523+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:50.916780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T15:16:35.006816+0000","last_change":"2024-09-21T15:16:35.006816+0000","last_active":"2024-09-21T15:16:35.006816+0000","last_peered":"2024-09-21T15:16:35.006816+0000","last_clean":"2024-09-21T15:16:35.006816+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T15:16:35.006816+0000","last_undegraded":"2024-09-21T15:16:35.006816+0000","last_fullsized":"2024-09-21T15:16:35.006816+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":44,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-21T15:16:29.965241+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:16:29.965241+0000","last_peered":"2024-09-21T15:16:29.965241+0000","last_clean":"2024-09-21T15:16:29.965241+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:16:29.965241+0000","last_undegraded":"2024-09-21T15:16:29.965241+0000","last_fullsized":"2024-09-21T15:16:29.965241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":565248,"data_stored":771568,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.82899999999999996}]},{"osd":1,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.873}]},{"osd":2,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.66100000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.76900000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.57599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":507904,"data_stored":181200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.78000000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.68899999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.85899999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.62}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":589824,"data_stored":780871,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.72699999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.86099999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.79200000000000004}]},{"osd":5,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.91400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31644,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340196,"statfs":{"total":96636764160,"available":96604360704,"internally_reserved":0,"allocated":536576,"data_stored":190522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.65700000000000003}]},{"osd":2,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.501},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.80600000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.91800000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.84499999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.96099999999999997}]}]},{"osd":4,"up_from":0,"seq":51539607573,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":51539607571,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:16:36.381 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:36.381 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:36.381 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:36.381 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:36.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:36.411 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:36.409+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:36.442 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:36.462 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:36.486 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:36.503 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:36.503 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:36.504 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:36.504 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:36.504 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:36.508 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:36.512 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:36.513+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:36.513 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:36.530 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:36.554 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:36.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:36.613 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:36.608+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:36.617 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:36.617 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:36.627 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:36.714 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:36.713+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:36.729 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:36.729 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:36.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:36.815 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:36.813+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:36.889 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:36.889 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:36.899 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:36.916 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:36.913+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:37.013 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.013 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.016 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:37.013+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:37.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:37.117 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:37.112+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:37.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:37.216+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:37.301 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.301 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:37.319 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:37.317+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:37.412 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.413 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.420 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:37.417+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:37.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:37.521 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:37.521+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:37.528 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.528 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:37.621 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:37.616+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:37.653 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.653 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:37.722 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:37.720+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:37.770 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.770 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:37.823 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:37.820+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:37.924 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:37.920+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:37.933 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:37.933 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:37.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:38.025 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:38.020+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:38.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:16:38.059 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:38.059 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:38.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:38.074 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false rbd-nbd 2024-09-21T15:16:38.074 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd-nbd 2024-09-21T15:16:38.125 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:38.125+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:38.132 INFO:tasks.workunit.client.0.smithi146.stderr:rbd-nbd: must specify command 2024-09-21T15:16:38.134 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:38.134 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-09-21T15:16:38.134 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd-nbd INVALIDCMD 2024-09-21T15:16:38.167 INFO:tasks.workunit.client.0.smithi146.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-09-21T15:16:38.168 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:38.169 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:38.171 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -ne 0 ']' 2024-09-21T15:16:38.171 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd17425 2024-09-21T15:16:38.171 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd map testrbdnbd17425 2024-09-21T15:16:38.222 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:38.222 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:38.226 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:38.224+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:38.232 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:38.311 INFO:tasks.workunit.client.0.smithi146.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-09-21T15:16:38.314 INFO:tasks.workunit.client.0.smithi146.stderr:rbd-nbd: failed to find unused device 2024-09-21T15:16:38.327 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:38.325+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:38.329 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-21T15:16:38.337 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:38.337 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-21T15:16:38.337 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-21T15:16:38.337 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:38.337 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:38.339 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:38.342 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:38.342 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:38.342 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-09-21T15:16:38.350 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:38.350 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:38.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:38.428 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:38.424+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:38.466 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:38.466 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:38.466 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-21T15:16:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd17425 2024-09-21T15:16:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd17425 2024-09-21T15:16:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:38.472 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:38.474 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:38.474 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd-nbd 2024-09-21T15:16:38.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:38.476 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-21T15:16:38.476 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:38.476 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd17425 2024-09-21T15:16:38.529 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:38.529+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:38.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:38.584 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:38.585 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:38.594 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:38.630 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:38.629+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:38.698 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:38.698 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:38.710 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:38.722 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:38.731 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:38.728+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:38.744 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:38.766 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:38.790 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:38.813 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:38.814 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:38.815 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:38.815 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:38.815 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:38.815 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:38.825 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:38.832 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:38.829+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:38.836 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:38.839 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:38.839 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:38.839 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:38.839 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:38.839 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:38.839 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.114256181000002, 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:38.933 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.104699597000007, 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:38.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:38.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:38.941 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.431478172999999, 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:38.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:38.945 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:38.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:38.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:38.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:38.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:38.946 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:38.946 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:38.933+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:38.947 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:39.034 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:39.033+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:39.063 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:16:39.098 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:39.098 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:39.098 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:39.098 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:39.106 INFO:tasks.workunit.client.0.smithi146.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-09-21T15:16:39.107 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:39.118 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:39.118 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-09-21T15:16:39.118 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd --format INVALID list 2024-09-21T15:16:39.136 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:39.132+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:39.153 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: the argument for option '--format' is invalid 2024-09-21T15:16:39.155 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:39.155 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-09-21T15:16:39.211 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:39.211 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:39.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:39.223 INFO:tasks.workunit.client.0.smithi146.stdout:[] 2024-09-21T15:16:39.228 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd --format xml list 2024-09-21T15:16:39.236 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:39.237+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:39.299 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:16:39.300 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:39.300 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:39.302 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:39.302 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:16:39.305 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:16:39.305 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:39.305 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:16:39.325 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:39.325 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:39.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:39.325 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:39.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.325 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:23 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'13, mlcod=15'11)", 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.429209395000001, 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050072939999999998, 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894385+0000", 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894383+0000", 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894394+0000", 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.081e-05 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894419+0000", 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4355999999999999e-05 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894430+0000", 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.119e-05 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894524+0000", 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4493999999999993e-05 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.894575+0000", 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0294000000000003e-05 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899307+0000", 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047323629999999998 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.899392+0000", 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5309999999999995e-05 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:33 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'19, mlcod=15'14)", 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.407045217000004, 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047415039999999997, 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.330 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916549+0000", 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916547+0000", 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916562+0000", 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4192999999999999e-05 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916589+0000", 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7305999999999999e-05 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916601+0000", 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2057e-05 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916685+0000", 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4067999999999995e-05 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.916774+0000", 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8933999999999999e-05 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921182+0000", 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0044084249999999997 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921291+0000", 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010830100000000001 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'15)", 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:39.334 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.405598874999995, 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0060199229999999999, 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917996+0000", 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.917994+0000", 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918006+0000", 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1939999999999999e-05 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918028+0000", 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2129000000000001e-05 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918039+0000", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0699999999999999e-05 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918122+0000", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3115000000000006e-05 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918152+0000", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9454999999999998e-05 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.923933+0000", 2024-09-21T15:16:39.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0057817049999999998 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924015+0000", 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2142000000000001e-05 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'15)", 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:39.338 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.404625319000004, 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.005115596, 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918969+0000", 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.918968+0000", 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919018+0000", 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0275000000000002e-05 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919038+0000", 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.946e-05 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919047+0000", 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919108+0000", 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0847999999999997e-05 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.919129+0000", 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1287000000000001e-05 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924021+0000", 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0048923170000000002 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.924085+0000", 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3368999999999996e-05 2024-09-21T15:16:39.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.402198459000005, 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:39.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.401490644000006, 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.328973118999997, 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.328180412999998, 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.327014012000006, 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.361 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:39.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:39.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'44, mlcod=15'39)", 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.325223151000003, 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063785480000000004, 2024-09-21T15:16:39.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998371+0000", 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998370+0000", 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998379+0000", 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7600000000000008e-06 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998405+0000", 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5792999999999999e-05 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998413+0000", 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4489999999999999e-06 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000582+0000", 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0021689840000000001 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.000622+0000", 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0151999999999998e-05 2024-09-21T15:16:39.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004723+0000", 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.004100851 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004750+0000", 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6639e-05 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:84 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'50)", 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.294606772999998, 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050213109999999997, 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:39.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028988+0000", 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.028986+0000", 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029060+0000", 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3437999999999997e-05 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029086+0000", 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6553000000000001e-05 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029107+0000", 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1083000000000001e-05 2024-09-21T15:16:39.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029188+0000", 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0668999999999996e-05 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.029220+0000", 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2274000000000003e-05 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.033925+0000", 2024-09-21T15:16:39.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0047049090000000002 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034009+0000", 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3714999999999994e-05 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.292638562999997, 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:39.374 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:86 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'56, mlcod=15'51)", 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:39.378 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.290885544000005, 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052999960000000004, 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032709+0000", 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032708+0000", 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032718+0000", 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0906e-05 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032742+0000", 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3397e-05 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032753+0000", 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032809+0000", 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5844000000000003e-05 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.032829+0000", 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0452e-05 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037978+0000", 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051486350000000004 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.038009+0000", 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1192999999999998e-05 2024-09-21T15:16:39.382 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:87 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'57, mlcod=15'52)", 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.289500903999993, 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0052640159999999998, 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034093+0000", 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034092+0000", 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034145+0000", 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307999999999999e-05 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034170+0000", 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4830000000000001e-05 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034185+0000", 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4423000000000001e-05 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034251+0000", 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6848999999999996e-05 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.034277+0000", 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5876000000000001e-05 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039308+0000", 2024-09-21T15:16:39.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0050310709999999998 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.039357+0000", 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9061999999999999e-05 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.088652451000002, 2024-09-21T15:16:39.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:39.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:39.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:106 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'68, mlcod=15'64)", 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.087887518000002, 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063246220000000002, 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.392 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235707+0000", 2024-09-21T15:16:39.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235705+0000", 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235759+0000", 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3335999999999999e-05 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235774+0000", 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4926000000000001e-05 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235784+0000", 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8069999999999994e-06 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236483+0000", 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00069900799999999999 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236524+0000", 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1196000000000003e-05 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242012+0000", 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0054881519999999996 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242031+0000", 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9616999999999999e-05 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.396 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.086877129000001, 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:39.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:39.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:108 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'70, mlcod=15'64)", 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:39.402 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.086016721999997, 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0061737859999999997, 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237578+0000", 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237576+0000", 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237590+0000", 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4117000000000001e-05 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237617+0000", 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6749000000000001e-05 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.237649+0000", 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1606000000000002e-05 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239471+0000", 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001822747 2024-09-21T15:16:39.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239508+0000", 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6628999999999998e-05 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243678+0000", 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041696370000000003 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243751+0000", 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3813999999999995e-05 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.084686851000001, 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.407 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:39.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:39.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:110 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'72, mlcod=15'65)", 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.083227432000001, 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053473430000000001, 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:39.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240367+0000", 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240366+0000", 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240380+0000", 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4486e-05 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240397+0000", 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6546e-05 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240407+0000", 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0127e-05 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242117+0000", 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017101989999999999 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242176+0000", 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9462000000000001e-05 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245665+0000", 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034887400000000002 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:39.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245714+0000", 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9196000000000001e-05 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:39.416 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:39.417 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:39.417+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:39.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:39.462 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:39.462 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:16:39.462 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:16:39.462 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:16:39.463 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:16:39.463 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:16:39.519 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:39.516+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:39.538 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:39.538 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:39.539 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:39.539 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:39.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=17722 2024-09-21T15:16:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 17722 2024-09-21T15:16:39.557 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 17722 -C rbd-nbd 2024-09-21T15:16:39.569 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:39.570 INFO:tasks.workunit.client.0.smithi146.stdout: 17722 ? 00:00:00 rbd-nbd 2024-09-21T15:16:39.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17425 2024-09-21T15:16:39.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17425 2024-09-21T15:16:39.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:39.570 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:39.572 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:39.572 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd-nbd 2024-09-21T15:16:39.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-21T15:16:39.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:39.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17425 2024-09-21T15:16:39.619 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:39.616+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:39.659 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:39.659 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:39.659 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:39.659 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:39.659 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:39.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:39.675 INFO:tasks.workunit.client.0.smithi146.stderr:rbd-nbd: the device /dev/nbd0 is busy 2024-09-21T15:16:39.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:39.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ dev1=/dev/nbd0 2024-09-21T15:16:39.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 17722 2024-09-21T15:16:39.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:16:39.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=17722 2024-09-21T15:16:39.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:39.694 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:39.694 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:39.694 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:39.695 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:39.697 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:39.697 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:39.697 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:39.720 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:39.716+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:39.797 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:39.798 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^17722\b' 2024-09-21T15:16:39.798 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^17722\b' 2024-09-21T15:16:39.821 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:39.821+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.002303148999999, 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050810819999999998, 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821571+0000", 2024-09-21T15:16:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821568+0000", 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821585+0000", 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6597000000000001e-05 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821619+0000", 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4048999999999999e-05 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821645+0000", 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6483e-05 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.821799+0000", 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015363599999999999 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822170+0000", 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000371312 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.822485+0000", 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031455300000000002 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.824366+0000", 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001880633 2024-09-21T15:16:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826571+0000", 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022050580000000002 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826617+0000", 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6867999999999999e-05 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:23.826652+0000", 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4592999999999999e-05 2024-09-21T15:16:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4208.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.992148216999993, 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024520610000000002, 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4208", 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.67:0/1374558260", 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831726+0000", 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831723+0000", 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831749+0000", 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6222999999999999e-05 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831779+0000", 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9683e-05 2024-09-21T15:16:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831802+0000", 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3129000000000002e-05 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831899+0000", 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7280000000000004e-05 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.831997+0000", 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7585999999999997e-05 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.832194+0000", 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019738199999999999 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.833469+0000", 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001275188 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834095+0000", 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062557900000000002 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834142+0000", 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6594999999999997e-05 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:24.834178+0000", 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6418999999999998e-05 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4986.0:61 2.5 e50/48 2:a72945ab:ns::rbd_object_map.137aae78387e:head v 50'25, mlcod=50'23)", 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.271029+0000", 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5528453520000003, 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012974379999999999, 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271029+0000", 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271029+0000", 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271027+0000", 2024-09-21T15:16:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271048+0000", 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0746000000000001e-05 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271080+0000", 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2938999999999998e-05 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271093+0000", 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2833000000000001e-05 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271202+0000", 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108365 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.271244+0000", 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2738999999999999e-05 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.272251+0000", 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010064080000000001 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.272326+0000", 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5403999999999998e-05 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4986.0:62 2.6 2:6e955b78:ns::rbd_header.137aae78387e:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.273065+0000", 2024-09-21T15:16:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5508089959999998, 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020092360000000002, 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4986", 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1464530257", 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273065+0000", 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273065+0000", 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273062+0000", 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273079+0000", 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273112+0000", 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2576000000000001e-05 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273128+0000", 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5702999999999999e-05 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273228+0000", 2024-09-21T15:16:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100721 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273380+0000", 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151197 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.273650+0000", 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026987700000000002 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274923+0000", 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012735820000000001 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.274936+0000", 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3040000000000001e-05 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275016+0000", 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0224999999999998e-05 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275074+0000", 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7969e-05 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4986.0:64 2.6 2:6e955b78:ns::rbd_header.137aae78387e:head [watch unwatch cookie 139885407133504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.275578+0000", 2024-09-21T15:16:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5482964299999997, 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026242650000000002, 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4986", 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1464530257", 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 64 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275578+0000", 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275578+0000", 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275577+0000", 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275586+0000", 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6320000000000003e-06 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275598+0000", 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1372000000000001e-05 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.275608+0000", 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0185e-05 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276265+0000", 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065655399999999995 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276412+0000", 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014707699999999999 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276640+0000", 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227879 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277727+0000", 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001087066 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277802+0000", 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4987999999999994e-05 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277941+0000", 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013944399999999999 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278202+0000", 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260931 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4986.0:65 2.6 2:6e955b78:ns::rbd_header.137aae78387e:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:34.276487+0000", 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5473867219999997, 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020420500000000001, 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4986", 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1464530257", 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 65 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276487+0000", 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276487+0000", 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276486+0000", 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276494+0000", 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0080000000000002e-06 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276520+0000", 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5616e-05 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.276530+0000", 2024-09-21T15:16:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277062+0000", 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053193000000000001 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for rw locks", 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.277335+0000", 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000272417 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278281+0000", 2024-09-21T15:16:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000946702 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278401+0000", 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119368 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:34.278529+0000", 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012877400000000001 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5010.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:36.027223+0000", 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.7966512410000002, 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014744718, 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5010", 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1277780976", 2024-09-21T15:16:39.858 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027223+0000", 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027223+0000", 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027221+0000", 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027233+0000", 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1965e-05 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027293+0000", 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9789e-05 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027313+0000", 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0849999999999999e-05 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027414+0000", 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010073699999999999 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027421+0000", 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0040000000000002e-06 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.029408+0000", 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019872589999999999 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.029422+0000", 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3821e-05 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.036546+0000", 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071233379999999999 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.036749+0000", 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020337299999999999 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.036917+0000", 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000168405 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041642+0000", 2024-09-21T15:16:39.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047242209999999998 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041839+0000", 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019750799999999999 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041907+0000", 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7414000000000002e-05 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041968+0000", 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1093000000000006e-05 2024-09-21T15:16:39.864 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4960.0:15 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [watch unwatch cookie 139626098560224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.109519+0000", 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.71435536399999999, 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031110830000000002, 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4960", 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3738677174", 2024-09-21T15:16:39.865 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 15 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109519+0000", 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109519+0000", 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109517+0000", 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109528+0000", 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1464999999999999e-05 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109561+0000", 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2474000000000001e-05 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109577+0000", 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6665000000000001e-05 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109705+0000", 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012768299999999999 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.109822+0000", 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116542 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.110004+0000", 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000182366 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.111419+0000", 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014146720000000001 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.112481+0000", 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001062141 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.112550+0000", 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9381999999999994e-05 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.112630+0000", 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9563999999999999e-05 2024-09-21T15:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4978.0:3 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.412430+0000", 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.41144394200000001, 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001119999, 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4978", 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1374013348", 2024-09-21T15:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412430+0000", 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412430+0000", 2024-09-21T15:16:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412428+0000", 2024-09-21T15:16:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:39.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412441+0000", 2024-09-21T15:16:39.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3101999999999999e-05 2024-09-21T15:16:39.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412466+0000", 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4692999999999999e-05 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412487+0000", 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1233000000000002e-05 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412649+0000", 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016158000000000001 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.412891+0000", 2024-09-21T15:16:39.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000242215 2024-09-21T15:16:39.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.413550+0000", 2024-09-21T15:16:39.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065932199999999995 2024-09-21T15:16:39.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.878 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.878 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4978.0:5 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call,call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.879 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.414781+0000", 2024-09-21T15:16:39.879 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.409093504, 2024-09-21T15:16:39.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081667199999999995, 2024-09-21T15:16:39.879 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.879 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.879 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.879 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4978", 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1374013348", 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.414781+0000", 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.414781+0000", 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.414779+0000", 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.414788+0000", 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3430000000000002e-06 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.414809+0000", 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0302999999999999e-05 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.414820+0000", 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1090000000000001e-05 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.414935+0000", 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114864 2024-09-21T15:16:39.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.415167+0000", 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023197800000000001 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.415597+0000", 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043077700000000003 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4978.0:7 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.417212+0000", 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.40666168699999999, 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001344514, 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4978", 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1374013348", 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417212+0000", 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417212+0000", 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417211+0000", 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417221+0000", 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6930000000000002e-06 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417242+0000", 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1699e-05 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417252+0000", 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4269999999999997e-06 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417359+0000", 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010729400000000001 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.417606+0000", 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024665499999999998 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.418557+0000", 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095114899999999998 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4978.0:8 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call,call rbd.parent_overlap_get in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.418916+0000", 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.40495818700000002, 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001128228, 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4978", 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1374013348", 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 8 2024-09-21T15:16:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.418916+0000", 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.418916+0000", 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.418915+0000", 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.418923+0000", 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2600000000000005e-06 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.418943+0000", 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9735999999999999e-05 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.418952+0000", 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1230000000000006e-06 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.419044+0000", 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1904999999999995e-05 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.419274+0000", 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023039100000000001 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.420044+0000", 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077005300000000004 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4978.0:11 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.422891+0000", 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.40098269399999997, 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076434599999999995, 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.892 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4978", 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1374013348", 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 11 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.422891+0000", 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.422891+0000", 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.422889+0000", 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.422903+0000", 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3743e-05 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.422936+0000", 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3266000000000003e-05 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.422952+0000", 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6042999999999999e-05 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.423076+0000", 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123471 2024-09-21T15:16:39.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.423323+0000", 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024749 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.423656+0000", 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033245300000000002 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4978.0:12 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [watch watch cookie 139677638085200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.424230+0000", 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.39964423399999999, 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027970650000000001, 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4978", 2024-09-21T15:16:39.897 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1374013348", 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 12 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424230+0000", 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424230+0000", 2024-09-21T15:16:39.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424228+0000", 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424256+0000", 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8240000000000001e-05 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424289+0000", 2024-09-21T15:16:39.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2872e-05 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424303+0000", 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.42e-05 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424378+0000", 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4777999999999995e-05 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424490+0000", 2024-09-21T15:16:39.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112052 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.424685+0000", 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000195393 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.425562+0000", 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087660400000000001 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.426875+0000", 2024-09-21T15:16:39.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013133109999999999 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.426940+0000", 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4620000000000001e-05 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.427027+0000", 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7068000000000001e-05 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4978.0:14 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.428536+0000", 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.39533859700000001, 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083672100000000001, 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4978", 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1374013348", 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 14 2024-09-21T15:16:39.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428536+0000", 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428536+0000", 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428534+0000", 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428544+0000", 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0642000000000001e-05 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428568+0000", 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3280000000000001e-05 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428580+0000", 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2602999999999999e-05 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428703+0000", 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123151 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.428981+0000", 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027744999999999999 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.429372+0000", 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039130799999999999 2024-09-21T15:16:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4987.0:3 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.642222+0000", 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.18165257700000001, 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00074516000000000003, 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4987", 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2607033757", 2024-09-21T15:16:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642222+0000", 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642222+0000", 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642219+0000", 2024-09-21T15:16:39.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642234+0000", 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4375999999999999e-05 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642255+0000", 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1276000000000001e-05 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642270+0000", 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4687e-05 2024-09-21T15:16:39.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642376+0000", 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106937 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642542+0000", 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000165217 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.642967+0000", 2024-09-21T15:16:39.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042499999999999998 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4987.0:7 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.646412+0000", 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.177461648, 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011223520000000001, 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4987", 2024-09-21T15:16:39.911 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2607033757", 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 7 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646412+0000", 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646412+0000", 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646411+0000", 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646423+0000", 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1895999999999999e-05 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646448+0000", 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5715999999999999e-05 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646462+0000", 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.379e-05 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646554+0000", 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1960999999999996e-05 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.646702+0000", 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148026 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.647535+0000", 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083267599999999997 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4987.0:8 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [call,call rbd.parent_overlap_get in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.647923+0000", 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.17595101199999999, 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010507349999999999, 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4987", 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2607033757", 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 8 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.647923+0000", 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.647923+0000", 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.647922+0000", 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.647930+0000", 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5909999999999998e-06 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.647943+0000", 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2828e-05 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.647949+0000", 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7259999999999999e-06 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.648069+0000", 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119962 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.648318+0000", 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024924099999999998 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.648974+0000", 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065607600000000001 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4987.0:12 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [watch watch cookie 140608840681984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.919 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.652269+0000", 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.17160523, 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026366950000000001, 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4987", 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2607033757", 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 12 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652269+0000", 2024-09-21T15:16:39.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652269+0000", 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652267+0000", 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652277+0000", 2024-09-21T15:16:39.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6399999999999992e-06 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652298+0000", 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0639999999999999e-05 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652309+0000", 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1036000000000001e-05 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652374+0000", 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4919000000000006e-05 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652455+0000", 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1662999999999994e-05 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.652606+0000", 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015055 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.653552+0000", 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000946068 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.654794+0000", 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001241811 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.654845+0000", 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1020999999999998e-05 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.654906+0000", 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0794000000000001e-05 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4987.0:15 2.6 2:6fafaf5b:::rbd_header.13743413a48b:head [watch unwatch cookie 140608840681984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:39.678565+0000", 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.14530910799999999, 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029084810000000001, 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:39.925 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4987", 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2607033757", 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 15 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678565+0000", 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:39.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678565+0000", 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678562+0000", 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999967 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678580+0000", 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8598999999999999e-05 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678621+0000", 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0926000000000001e-05 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678640+0000", 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8243000000000001e-05 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678753+0000", 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113112 2024-09-21T15:16:39.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.678905+0000", 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015197400000000001 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.679115+0000", 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021074099999999999 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.680135+0000", 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001019552 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.681310+0000", 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011754549999999999 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.681382+0000", 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1710000000000003e-05 2024-09-21T15:16:39.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:39.681473+0000", 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1395000000000002e-05 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:39.931 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:39.933 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:39.933 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:39.933 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *17722\b' 2024-09-21T15:16:39.933 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *17722\b' 2024-09-21T15:16:39.934 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:39.934 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:39.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:39.934 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:39.933+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:40.034 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:40.033+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:40.053 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:40.053 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:40.053 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:40.053 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:40.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:40.135 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:40.132+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:40.170 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:40.170 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:40.170 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:40.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:40.235 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:40.233+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:40.294 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.510820469999999, 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.377010100000007, 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017153211000000002, 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916031+0000", 2024-09-21T15:16:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916030+0000", 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916041+0000", 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1226000000000001e-05 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916058+0000", 2024-09-21T15:16:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6551000000000001e-05 2024-09-21T15:16:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916076+0000", 2024-09-21T15:16:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196999999999999e-05 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.923738+0000", 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076618199999999997 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924051+0000", 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000313033 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924276+0000", 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022544899999999999 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.928804+0000", 2024-09-21T15:16:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045275999999999997 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933045+0000", 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042412559999999997 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933132+0000", 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6860999999999994e-05 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933185+0000", 2024-09-21T15:16:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2729999999999998e-05 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.376950872999998, 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017314743, 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916091+0000", 2024-09-21T15:16:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916090+0000", 2024-09-21T15:16:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916101+0000", 2024-09-21T15:16:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1549000000000001e-05 2024-09-21T15:16:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916118+0000", 2024-09-21T15:16:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6578999999999998e-05 2024-09-21T15:16:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.916144+0000", 2024-09-21T15:16:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6310999999999999e-05 2024-09-21T15:16:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.924995+0000", 2024-09-21T15:16:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088507959999999993 2024-09-21T15:16:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925138+0000", 2024-09-21T15:16:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014346199999999999 2024-09-21T15:16:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.925372+0000", 2024-09-21T15:16:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023338499999999999 2024-09-21T15:16:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.931525+0000", 2024-09-21T15:16:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061536179999999996 2024-09-21T15:16:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933252+0000", 2024-09-21T15:16:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263370000000001 2024-09-21T15:16:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933323+0000", 2024-09-21T15:16:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1443e-05 2024-09-21T15:16:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.933405+0000", 2024-09-21T15:16:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2303000000000001e-05 2024-09-21T15:16:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.262628045, 2024-09-21T15:16:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01367899, 2024-09-21T15:16:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 88 2024-09-21T15:16:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030413+0000", 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030417+0000", 2024-09-21T15:16:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1810000000000003e-06 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030422+0000", 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4560000000000002e-06 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.030425+0000", 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000004e-06 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038695+0000", 2024-09-21T15:16:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082695340000000003 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.038851+0000", 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156059 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.039012+0000", 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016049599999999999 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.040398+0000", 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013868680000000001 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.043975+0000", 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035764640000000001 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044046+0000", 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1561000000000001e-05 2024-09-21T15:16:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.044092+0000", 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6004000000000002e-05 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.055060183999998, 2024-09-21T15:16:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014644463999999999, 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 109 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237981+0000", 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.237980+0000", 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238108+0000", 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012861700000000001 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238129+0000", 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0990000000000001e-05 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238140+0000", 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0297e-05 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241779+0000", 2024-09-21T15:16:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003638914 2024-09-21T15:16:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.241896+0000", 2024-09-21T15:16:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011769000000000001 2024-09-21T15:16:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.242121+0000", 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022516800000000001 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247096+0000", 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049745409999999999 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252453+0000", 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053567240000000002 2024-09-21T15:16:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252573+0000", 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120667 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252626+0000", 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2438999999999997e-05 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054890244000006, 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014880765000000001, 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 110 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238151+0000", 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238150+0000", 2024-09-21T15:16:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238157+0000", 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8059999999999997e-06 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238166+0000", 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-06 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238173+0000", 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1770000000000001e-06 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243328+0000", 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051550490000000001 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243480+0000", 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000152039 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.243630+0000", 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149734 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248554+0000", 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049242139999999997 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252786+0000", 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042314309999999999 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252913+0000", 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012753499999999999 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253032+0000", 2024-09-21T15:16:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118653 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054861344000003, 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015142765000000001, 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 111 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238180+0000", 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238185+0000", 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4659999999999998e-06 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238193+0000", 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2130000000000008e-06 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238200+0000", 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7329999999999997e-06 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244440+0000", 2024-09-21T15:16:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062401369999999998 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244591+0000", 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150951 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.244816+0000", 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022449199999999999 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.248589+0000", 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037738749999999999 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253139+0000", 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045500059999999997 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253273+0000", 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013360499999999999 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253323+0000", 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9907000000000002e-05 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054809868999996, 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015259925000000001, 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 112 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238232+0000", 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238230+0000", 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238372+0000", 2024-09-21T15:16:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014151100000000001 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238389+0000", 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6875e-05 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238398+0000", 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245548+0000", 2024-09-21T15:16:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071493499999999996 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245674+0000", 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012637999999999999 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.245847+0000", 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017249799999999999 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249111+0000", 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032639969999999998 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253396+0000", 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042854160000000002 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253453+0000", 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253492+0000", 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8837000000000001e-05 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054631178999998, 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015255476, 2024-09-21T15:16:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 113 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238410+0000", 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238409+0000", 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238416+0000", 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9940000000000003e-06 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238466+0000", 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9707999999999999e-05 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238478+0000", 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1663e-05 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247187+0000", 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087092260000000005 2024-09-21T15:16:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247314+0000", 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012676399999999999 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.247475+0000", 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016153700000000001 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.252348+0000", 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004872726 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253556+0000", 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012079860000000001 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253622+0000", 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5624999999999996e-05 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253666+0000", 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4141000000000003e-05 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054548370999996, 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01532826, 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 114 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238493+0000", 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238492+0000", 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238502+0000", 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0253e-05 2024-09-21T15:16:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238594+0000", 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1741000000000001e-05 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238618+0000", 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4168999999999998e-05 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249155+0000", 2024-09-21T15:16:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010536703999999999 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249401+0000", 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024568499999999999 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.249610+0000", 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020972800000000001 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253053+0000", 2024-09-21T15:16:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003442409 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253735+0000", 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068195899999999999 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253784+0000", 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9333999999999999e-05 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253821+0000", 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7338000000000001e-05 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054390514999994, 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015358658000000001, 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 115 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238651+0000", 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238650+0000", 2024-09-21T15:16:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238746+0000", 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5501000000000006e-05 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238759+0000", 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.324e-05 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238767+0000", 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2260000000000005e-06 2024-09-21T15:16:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250309+0000", 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011541612999999999 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250491+0000", 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018238 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.250869+0000", 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037813000000000001 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253841+0000", 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029721610000000001 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253906+0000", 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5234000000000006e-05 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.253969+0000", 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2275999999999998e-05 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254010+0000", 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0890000000000003e-05 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054265803999996, 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017704856000000001, 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 116 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238776+0000", 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238775+0000", 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238781+0000", 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3899999999999998e-06 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238792+0000", 2024-09-21T15:16:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0743e-05 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238799+0000", 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.063e-06 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254069+0000", 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015270558 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254169+0000", 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9691999999999994e-05 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254308+0000", 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138634 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255866+0000", 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001558577 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256392+0000", 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052612000000000004 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256442+0000", 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9345999999999998e-05 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256481+0000", 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8720000000000002e-05 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.054235751999997, 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018045928999999999, 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 117 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238806+0000", 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238805+0000", 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238812+0000", 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0199999999999997e-06 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238820+0000", 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.238827+0000", 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8859999999999998e-06 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.254834+0000", 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016006807000000001 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255009+0000", 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174468 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.255339+0000", 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033008100000000003 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256499+0000", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011604650000000001 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256754+0000", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025443299999999998 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256806+0000", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2568000000000002e-05 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.256852+0000", 2024-09-21T15:16:40.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5577000000000001e-05 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.608027823, 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013767102, 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 135 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685014+0000", 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685013+0000", 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685020+0000", 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.08e-06 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685035+0000", 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228999999999999e-05 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685046+0000", 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.114e-05 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693634+0000", 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085881689999999997 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693779+0000", 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014467000000000001 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.693999+0000", 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021951499999999999 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697500+0000", 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035015799999999998 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698700+0000", 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011999619999999999 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698748+0000", 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8044000000000003e-05 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698781+0000", 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2533000000000002e-05 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.607987214000005, 2024-09-21T15:16:40.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013961045, 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 136 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685054+0000", 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685060+0000", 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685082+0000", 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2388999999999999e-05 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685098+0000", 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6053e-05 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694770+0000", 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096722160000000008 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.694892+0000", 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012193000000000001 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695019+0000", 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126668 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698800+0000", 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003780903 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698907+0000", 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107594 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.698951+0000", 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3429e-05 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699015+0000", 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4626000000000001e-05 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.607935802, 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017857213, 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 137 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685106+0000", 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685105+0000", 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685255+0000", 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015037300000000001 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685269+0000", 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3773e-05 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685277+0000", 2024-09-21T15:16:40.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0499999999999992e-06 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695776+0000", 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010498407 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695858+0000", 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2347000000000003e-05 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.695977+0000", 2024-09-21T15:16:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119257 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702788+0000", 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068105040000000002 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702865+0000", 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7657999999999994e-05 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702911+0000", 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5330999999999997e-05 2024-09-21T15:16:40.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.702963+0000", 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2176000000000003e-05 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.607750839000005, 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017810072, 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 138 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685291+0000", 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685290+0000", 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685296+0000", 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.083e-06 2024-09-21T15:16:40.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685304+0000", 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685311+0000", 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697596+0000", 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012284489000000001 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697757+0000", 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016078300000000001 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.697993+0000", 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023625799999999999 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703005+0000", 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005011878 2024-09-21T15:16:40.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703017+0000", 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2238999999999999e-05 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703059+0000", 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1780000000000003e-05 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703101+0000", 2024-09-21T15:16:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1826000000000002e-05 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.607721307000006, 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018645333, 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 139 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.404 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685320+0000", 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685333+0000", 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3113e-05 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685348+0000", 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5656000000000001e-05 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685355+0000", 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9670000000000004e-06 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699078+0000", 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013722432999999999 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699159+0000", 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1621999999999999e-05 2024-09-21T15:16:40.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699280+0000", 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120872 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703169+0000", 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038885489999999998 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703893+0000", 2024-09-21T15:16:40.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072430300000000003 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703939+0000", 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5859999999999998e-05 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703966+0000", 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6571e-05 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.607678610999997, 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018682681, 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.410 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 140 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685363+0000", 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685362+0000", 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685485+0000", 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122924 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685529+0000", 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3806e-05 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685539+0000", 2024-09-21T15:16:40.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6490000000000003e-06 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699873+0000", 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014334858000000001 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.699958+0000", 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4606999999999998e-05 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.700077+0000", 2024-09-21T15:16:40.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119375 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703207+0000", 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031292429999999999 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703976+0000", 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076900799999999995 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704012+0000", 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6396000000000002e-05 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704046+0000", 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3467999999999999e-05 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.607436938999996, 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019729155000000002, 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 141 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:40.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685605+0000", 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685604+0000", 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685616+0000", 2024-09-21T15:16:40.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2067e-05 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685627+0000", 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0829000000000001e-05 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.685634+0000", 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703235+0000", 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017600482000000001 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703360+0000", 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125862 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.703473+0000", 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011294900000000001 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:40.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.704313+0000", 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083986200000000003 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705270+0000", 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095693500000000001 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705309+0000", 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8831999999999997e-05 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:18.705334+0000", 2024-09-21T15:16:40.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4723999999999999e-05 2024-09-21T15:16:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.421 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.421 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:40.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:16:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17425 2024-09-21T15:16:40.423 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd-nbd 2024-09-21T15:16:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd-nbd 2024-09-21T15:16:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:40.424 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd17425 2024-09-21T15:16:40.425 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:40.421+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:40.513 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:40.513 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-09-21T15:16:40.514 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:40.514 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep testrbdnbd17425 2024-09-21T15:16:40.523 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:40.521+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:40.549 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:40.549 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:40.549 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:40.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:40.596 INFO:tasks.workunit.client.0.smithi146.stdout:17820 rbd testrbdnbd17425 - /dev/nbd0 2024-09-21T15:16:40.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:16:40.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:16:40.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:16:40.597 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:16:40.597 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:16:40.624 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:40.620+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:40.670 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:40.670 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:40.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=17820 2024-09-21T15:16:40.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 17820 2024-09-21T15:16:40.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 17820 -C rbd-nbd 2024-09-21T15:16:40.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:40.689 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:40.689 INFO:tasks.workunit.client.0.smithi146.stdout: 17820 ? 00:00:00 rbd-nbd 2024-09-21T15:16:40.690 INFO:tasks.workunit.client.0.smithi146.stderr:++ dd if=/tmp/tmp.flUinkcHzA/data bs=1M 2024-09-21T15:16:40.691 INFO:tasks.workunit.client.0.smithi146.stderr:++ md5sum 2024-09-21T15:16:40.730 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:40.728+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:40.791 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.791 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.971184362000002, 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:40.796 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:40.801 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:40.819 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:40.819 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:40.820 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.128201 s, 523 MB/s 2024-09-21T15:16:40.821 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-09-21T15:16:40.821 INFO:tasks.workunit.client.0.smithi146.stderr:++ md5sum 2024-09-21T15:16:40.821 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:40.821 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:40.823 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:40.823 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which dd 2024-09-21T15:16:40.825 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/dd 2024-09-21T15:16:40.825 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:40.825 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-09-21T15:16:40.849 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:40.849+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:40.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:40.919 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:40.919 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:40.919 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:40.919 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:40.926 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:40.924+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:40.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:41.012 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:41.026 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:41.024+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:41.034 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:41.041 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:41.041 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:41.041 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:41.041 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:41.041 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:41.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:41.063 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:41.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:16:41.089 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:41.116 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:41.127 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:41.124+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:41.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:41.162 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:41.162 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:41.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:41.228 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:41.229+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:41.277 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:41.277 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:41.288 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:41.321 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:41.321 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:41.322 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.486915 s, 138 MB/s 2024-09-21T15:16:41.324 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' '898d8f40b6681f4fc13b04bd5118c73c -' = '898d8f40b6681f4fc13b04bd5118c73c -' ']' 2024-09-21T15:16:41.324 INFO:tasks.workunit.client.0.smithi146.stderr:+ dd if=/dev/urandom of=/tmp/tmp.flUinkcHzA/data bs=1M count=64 2024-09-21T15:16:41.329 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:41.324+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:41.332 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:41.332 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:41.332 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T15:16:41.332 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.427036 2024-09-21T15:16:41.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 0.42703641998754505 2024-09-21T15:16:41.393 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:41.393 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:41.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:41.430 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:41.428+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:41.511 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:41.511 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:41.521 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:41.564 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:41.565+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:41.630 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:41.630 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:41.632 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:41.628+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:41.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:41.734 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:41.733+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:41.749 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:41.749 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:41.759 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:41.835 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:41.836+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:41.892 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:41.892 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:41.892 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.550707 s, 122 MB/s 2024-09-21T15:16:41.893 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T15:16:41.893 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:41.893 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:41.895 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:41.896 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:16:41.899 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:16:41.899 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:41.899 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T15:16:41.910 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:41.910 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:41.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:41.970 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:41.968+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:42.037 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:42.032+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:42.046 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.046 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:42.077 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 0.427036 (8279862) 2024-09-21T15:16:42.078 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:16:42.138 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:42.136+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:42.157 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.157 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:42.239 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:42.236+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:42.272 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.272 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:42.340 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:42.336+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:42.392 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.392 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:42.441 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:42.441+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:42.541 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:42.541+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:42.558 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.558 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:42.642 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:42.640+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:42.682 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.682 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:42.743 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:42.741+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:42.795 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.796 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:42.843 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:42.841+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:42.911 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:42.911 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:42.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:42.944 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:42.940+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:43.027 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:43.027 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:43.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:43.045 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:43.040+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:43.146 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:43.145+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:43.160 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:43.160 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:43.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:43.171 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:43.247 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:43.244+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:43.283 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:43.326 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:43.346 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:43.348 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:43.344+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:43.370 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:43.384 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:43.384 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:43.394 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:43.405 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:43.424 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:43.446 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:43.449 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:43.444+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:43.499 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:43.499 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:43.499 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:43.499 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:43.509 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:43.549 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:43.549+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:43.616 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:43.616 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:43.617 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:43.617 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:43.617 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:43.626 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:43.650 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:43.648+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 79.909380862000006, 2024-09-21T15:16:43.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:43.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:43.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:43.732 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.899824277999997, 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:43.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:43.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:43.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:43.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.226602853999999, 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:43.740 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:43.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:43.751 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:43.748+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:43.853 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:43.853+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:43.866 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:43.867 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:43.867 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:43.867 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:43.878 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:43.954 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:43.953+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:43.987 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:43.987 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:43.987 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:43.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:44.054 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:44.052+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:44.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:16:44.101 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:44.101 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:44.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:44.101 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:36 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'22, mlcod=15'16)", 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.178533100999999, 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0079656970000000007, 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.102 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921396+0000", 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921395+0000", 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921404+0000", 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4789999999999996e-06 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921419+0000", 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5e-05 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921434+0000", 2024-09-21T15:16:44.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5279999999999999e-05 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921523+0000", 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9208000000000004e-05 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.921567+0000", 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4101999999999999e-05 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929281+0000", 2024-09-21T15:16:44.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0077137020000000002 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929362+0000", 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0462000000000003e-05 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:37 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'23, mlcod=15'17)", 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:44.106 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.177825286000001, 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0073139420000000004, 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922104+0000", 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922103+0000", 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922112+0000", 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6360000000000006e-06 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922129+0000", 2024-09-21T15:16:44.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6966999999999999e-05 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922144+0000", 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4919000000000001e-05 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922233+0000", 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8864999999999997e-05 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.922271+0000", 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7738999999999999e-05 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929366+0000", 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070949639999999996 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.929418+0000", 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2042000000000001e-05 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.110 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:67 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'41, mlcod=15'39)", 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.105307761000006, 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065650580000000004, 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994621+0000", 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994619+0000", 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994636+0000", 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6218999999999998e-05 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994665+0000", 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9663e-05 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994680+0000", 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5073e-05 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994764+0000", 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3158000000000006e-05 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.994804+0000", 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0275999999999999e-05 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001123+0000", 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0063190859999999998 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001186+0000", 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3382999999999999e-05 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:68 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'42, mlcod=15'39)", 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.104515054999993, 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064611629999999998, 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.115 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995414+0000", 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995413+0000", 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995424+0000", 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1097000000000001e-05 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995445+0000", 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0826e-05 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.995462+0000", 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7036000000000001e-05 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998275+0000", 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002813496 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.998317+0000", 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1841999999999997e-05 2024-09-21T15:16:44.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001773+0000", 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034558420000000002 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.001875+0000", 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000102268 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.103348654000001, 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:44.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:44.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:85 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'55, mlcod=15'50)", 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.068973205000006, 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070191250000000002, 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030956+0000", 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030954+0000", 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030965+0000", 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0856000000000001e-05 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030982+0000", 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6923e-05 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.030993+0000", 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031071+0000", 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8665000000000007e-05 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.031108+0000", 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6316999999999998e-05 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037908+0000", 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067999209999999996 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.037975+0000", 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7311999999999995e-05 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:105 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'67, mlcod=15'64)", 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 85.864987092999996, 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0070641560000000003, 2024-09-21T15:16:44.128 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234942+0000", 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234941+0000", 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.234957+0000", 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6066000000000002e-05 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235000+0000", 2024-09-21T15:16:44.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3328999999999998e-05 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235029+0000", 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9122999999999999e-05 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235342+0000", 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00031314599999999999 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.235417+0000", 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5135999999999993e-05 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.241948+0000", 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0065309440000000003 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.242006+0000", 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7812000000000001e-05 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.132 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:107 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'69, mlcod=15'64)", 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 85.863211770999996, 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069503020000000002, 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236717+0000", 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236715+0000", 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236733+0000", 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236762+0000", 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8045999999999999e-05 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.236785+0000", 2024-09-21T15:16:44.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3336000000000002e-05 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238365+0000", 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015798190000000001 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238422+0000", 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7225999999999997e-05 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243612+0000", 2024-09-21T15:16:44.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0051897469999999998 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.243668+0000", 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5918999999999998e-05 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:109 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'71, mlcod=15'65)", 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 85.861021492999996, 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0067466000000000002, 2024-09-21T15:16:44.137 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238908+0000", 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238906+0000", 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238977+0000", 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0594999999999997e-05 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.238997+0000", 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0273000000000001e-05 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.239009+0000", 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1956e-05 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240140+0000", 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011314719999999999 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.240200+0000", 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9898999999999997e-05 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245589+0000", 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0053887539999999999 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.245654+0000", 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.5166999999999995e-05 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:339 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.100456+0000", 2024-09-21T15:16:44.141 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9994734320000003, 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.006463496, 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 339 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100456+0000", 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100456+0000", 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100455+0000", 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100474+0000", 2024-09-21T15:16:44.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8749999999999998e-05 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100490+0000", 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6730000000000001e-05 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100497+0000", 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7299999999999999e-06 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.106013+0000", 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055158359999999997 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.106145+0000", 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013204300000000001 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.106919+0000", 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00077401 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:340 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.100503+0000", 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9994260320000001, 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0072698520000000003, 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 340 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100503+0000", 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100503+0000", 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100502+0000", 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100508+0000", 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100515+0000", 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5460000000000003e-06 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100522+0000", 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2400000000000004e-06 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.106964+0000", 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0064423830000000003 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107089+0000", 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012485999999999999 2024-09-21T15:16:44.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107773+0000", 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00068405600000000005 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:341 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9994019390000002, 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0082113480000000003, 2024-09-21T15:16:44.150 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 341 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100526+0000", 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100530+0000", 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0460000000000001e-06 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100545+0000", 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5733999999999999e-05 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100552+0000", 2024-09-21T15:16:44.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107817+0000", 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0072653520000000001 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107946+0000", 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012907000000000001 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.108738+0000", 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079230900000000005 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:342 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102475+0000", 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9974537259999998, 2024-09-21T15:16:44.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0072163549999999998, 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 342 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102475+0000", 2024-09-21T15:16:44.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102475+0000", 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102473+0000", 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102485+0000", 2024-09-21T15:16:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1362999999999999e-05 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102515+0000", 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9920000000000002e-05 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102534+0000", 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9446999999999999e-05 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.108780+0000", 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.006245942 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.108916+0000", 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136334 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.109692+0000", 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00077524600000000005 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:343 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9973806199999999, 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081198239999999994, 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 343 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102552+0000", 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6360000000000002e-06 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102584+0000", 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1813999999999997e-05 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102593+0000", 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6430000000000001e-06 2024-09-21T15:16:44.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.109769+0000", 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0071766089999999996 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.109886+0000", 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011657799999999999 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110668+0000", 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078233000000000003 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.163 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9973294429999999, 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.164 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9972783999999999, 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:358 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118212+0000", 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.981716627, 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.006840128, 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 358 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118212+0000", 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118212+0000", 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118212+0000", 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118216+0000", 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.25e-06 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118227+0000", 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0737000000000001e-05 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118237+0000", 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0913e-05 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.124444+0000", 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0062067440000000001 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.124524+0000", 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9563999999999999e-05 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125053+0000", 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00052871700000000005 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.175 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:359 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118245+0000", 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9816844740000001, 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075318529999999998, 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 359 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118245+0000", 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118245+0000", 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118244+0000", 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118247+0000", 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.49e-06 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118259+0000", 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1226999999999999e-05 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118265+0000", 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0159999999999997e-06 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125105+0000", 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.006840278 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125189+0000", 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3762000000000002e-05 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125776+0000", 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058772999999999996 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9816588570000002, 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.183 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.981632104, 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:44.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.187 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:44.188 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:44.188 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:44.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:44.190 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:44.184+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:44.289 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:44.288+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:44.295 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:44.295 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:44.295 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:44.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:44.390 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:44.389+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:44.467 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:44.467 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:44.467 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:44.467 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:44.467 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:44.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:44.491 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:44.488+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:44.592 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:44.588+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:44.602 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:44.602 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:44.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5010.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:36.027223+0000", 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.5749224030000004, 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014744718, 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5010", 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1277780976", 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027223+0000", 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027223+0000", 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027221+0000", 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027233+0000", 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1965e-05 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027293+0000", 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9789e-05 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027313+0000", 2024-09-21T15:16:44.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0849999999999999e-05 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027414+0000", 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010073699999999999 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.027421+0000", 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0040000000000002e-06 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.029408+0000", 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019872589999999999 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.029422+0000", 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3821e-05 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.036546+0000", 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071233379999999999 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.036749+0000", 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020337299999999999 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.036917+0000", 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000168405 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041642+0000", 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047242209999999998 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041839+0000", 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019750799999999999 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041907+0000", 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7414000000000002e-05 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:36.041968+0000", 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1093000000000006e-05 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:620 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.337372+0000", 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.26477343, 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030574480000000001, 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 620 2024-09-21T15:16:44.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.337372+0000", 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.337372+0000", 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.337370+0000", 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.338217+0000", 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084691300000000003 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.338235+0000", 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7645000000000001e-05 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.338243+0000", 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6119999999999999e-06 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.338279+0000", 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5848e-05 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.338337+0000", 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7302000000000001e-05 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.338430+0000", 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2882999999999998e-05 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.362289+0000", 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023859045999999998 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.367822+0000", 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055331850000000004 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.367894+0000", 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2192000000000003e-05 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.367946+0000", 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2408999999999999e-05 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:622 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.615 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.2273155069999999, 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033831148999999998, 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 622 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374818+0000", 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999886 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375659+0000", 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084095399999999999 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375676+0000", 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6812000000000001e-05 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375684+0000", 2024-09-21T15:16:44.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0900000000000005e-06 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375731+0000", 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6737000000000003e-05 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375824+0000", 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3350000000000006e-05 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375959+0000", 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013498500000000001 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.407620+0000", 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031660559999999997 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408527+0000", 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090730699999999999 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408600+0000", 2024-09-21T15:16:44.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3011999999999998e-05 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408661+0000", 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0785000000000002e-05 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:624 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.419667+0000", 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.182478613, 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031093571, 2024-09-21T15:16:44.621 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 624 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.419667+0000", 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.419667+0000", 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.419665+0000", 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420490+0000", 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082458800000000001 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420509+0000", 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9584000000000001e-05 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420519+0000", 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3030000000000005e-06 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420574+0000", 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4892999999999998e-05 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420656+0000", 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2313999999999998e-05 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420784+0000", 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127978 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.447966+0000", 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027181944999999999 2024-09-21T15:16:44.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450694+0000", 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027282130000000002 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450732+0000", 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7799999999999997e-05 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450760+0000", 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8479e-05 2024-09-21T15:16:44.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.144334848, 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.627 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:16:44.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:628 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.509904+0000", 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.092241515, 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029818144000000001, 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 628 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.509904+0000", 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.509904+0000", 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.509903+0000", 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.510741+0000", 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083824799999999999 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.510757+0000", 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6603e-05 2024-09-21T15:16:44.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.510766+0000", 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0309999999999998e-06 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.510803+0000", 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.510862+0000", 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9036999999999997e-05 2024-09-21T15:16:44.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.510946+0000", 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3967000000000004e-05 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.534760+0000", 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02381407 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539653+0000", 2024-09-21T15:16:44.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048922849999999997 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539694+0000", 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1609000000000001e-05 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539722+0000", 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7608999999999999e-05 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:630 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.546583+0000", 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.055561983, 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026791186000000002, 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 630 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.546583+0000", 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.546583+0000", 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.546581+0000", 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.547416+0000", 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000835205 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.547446+0000", 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9745999999999999e-05 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.547469+0000", 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3390000000000001e-05 2024-09-21T15:16:44.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.547537+0000", 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7418999999999999e-05 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.547631+0000", 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4061000000000006e-05 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.547767+0000", 2024-09-21T15:16:44.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013604200000000001 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.573158+0000", 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025390693999999998 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.573284+0000", 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000125967 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.573339+0000", 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5244000000000002e-05 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.573374+0000", 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5664999999999998e-05 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:632 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.580217+0000", 2024-09-21T15:16:44.644 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.021928239, 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032334979, 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 632 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.580217+0000", 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.580217+0000", 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.580216+0000", 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581059+0000", 2024-09-21T15:16:44.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084304599999999996 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581073+0000", 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4352000000000001e-05 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581079+0000", 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581158+0000", 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8170999999999994e-05 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581310+0000", 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015192 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581542+0000", 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023252900000000001 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.611597+0000", 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030054953999999998 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.612488+0000", 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000891236 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.612528+0000", 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9579000000000002e-05 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.612552+0000", 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4216999999999999e-05 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:634 2.6 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.624869+0000", 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.97727642999999997, 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033046593999999999, 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.650 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 634 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.624869+0000", 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.624869+0000", 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.624867+0000", 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625725+0000", 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085822800000000005 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625753+0000", 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7722999999999999e-05 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625766+0000", 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3436e-05 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625855+0000", 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8776000000000005e-05 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625927+0000", 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1983999999999994e-05 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.626023+0000", 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5970000000000004e-05 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.653662+0000", 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027639433000000001 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.657866+0000", 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042037840000000003 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.657896+0000", 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9522999999999998e-05 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.657915+0000", 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:16:44.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:636 2.6 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.665249+0000", 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.93689652000000001, 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028517948000000001, 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.656 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 636 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.665249+0000", 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.665249+0000", 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.665247+0000", 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.666289+0000", 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001042107 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.666322+0000", 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2775999999999999e-05 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.666336+0000", 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4439e-05 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.666392+0000", 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6134000000000002e-05 2024-09-21T15:16:44.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.666451+0000", 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8709000000000003e-05 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.666539+0000", 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7585999999999998e-05 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.691688+0000", 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025149417 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.693720+0000", 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020317669999999999 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.693748+0000", 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8107999999999999e-05 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.693767+0000", 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8913999999999999e-05 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.661 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:638 2.6 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.705706+0000", 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.89643899699999996, 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028100716000000001, 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 638 2024-09-21T15:16:44.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.705706+0000", 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.705706+0000", 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.705704+0000", 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.706693+0000", 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098879900000000006 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.706725+0000", 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1819000000000001e-05 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.706739+0000", 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4333e-05 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.706814+0000", 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5043000000000006e-05 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.706864+0000", 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9731000000000002e-05 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.706937+0000", 2024-09-21T15:16:44.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2947999999999993e-05 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.732701+0000", 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025763984 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.733755+0000", 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010541190000000001 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.733787+0000", 2024-09-21T15:16:44.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2016e-05 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.733807+0000", 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9944000000000001e-05 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:640 2.6 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.740998+0000", 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.86114764499999996, 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033725333000000003, 2024-09-21T15:16:44.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 640 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.740998+0000", 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.740998+0000", 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.740996+0000", 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.741954+0000", 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095838900000000003 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.741981+0000", 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6780000000000001e-05 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.741994+0000", 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3249e-05 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.742064+0000", 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0019999999999997e-05 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.742110+0000", 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6014999999999999e-05 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.742180+0000", 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9437999999999995e-05 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.764290+0000", 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022110396000000001 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.774674+0000", 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010383501999999999 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.774703+0000", 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9691000000000001e-05 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.774723+0000", 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9616e-05 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:650 2.5 2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.983799+0000", 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.61834645700000002, 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031828651999999999, 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 650 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.673 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.983799+0000", 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.983799+0000", 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.983797+0000", 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984706+0000", 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090949399999999995 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984762+0000", 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5365000000000003e-05 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984784+0000", 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1990000000000001e-05 2024-09-21T15:16:44.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984912+0000", 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012814400000000001 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.985060+0000", 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014847999999999999 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.985276+0000", 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021580899999999999 2024-09-21T15:16:44.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.014713+0000", 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029436859999999999 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.015565+0000", 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085232400000000001 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.015602+0000", 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6522e-05 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.015627+0000", 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5758e-05 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:652 2.5 2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.022787+0000", 2024-09-21T15:16:44.678 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.57935839600000005, 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033027978, 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 652 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.022787+0000", 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.022787+0000", 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.022785+0000", 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023706+0000", 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00092091600000000003 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023737+0000", 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0749e-05 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023758+0000", 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1124e-05 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023843+0000", 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5828000000000006e-05 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023993+0000", 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014912700000000001 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.024186+0000", 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019344499999999999 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.054957+0000", 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030771011000000001 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.055748+0000", 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079097 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.055778+0000", 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9859999999999999e-05 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.055815+0000", 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7001e-05 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:654 2.5 2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.063754+0000", 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.53839119199999996, 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030339727, 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 654 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.063754+0000", 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.063754+0000", 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.063753+0000", 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064593+0000", 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083975800000000002 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064610+0000", 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7723999999999998e-05 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064618+0000", 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1270000000000003e-06 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064670+0000", 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1249999999999999e-05 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064751+0000", 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1675000000000007e-05 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064851+0000", 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9281e-05 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.091830+0000", 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026979079 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.093950+0000", 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002120678 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.094037+0000", 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6975e-05 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.094094+0000", 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6416999999999999e-05 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:656 2.5 2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.106208+0000", 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.49593757300000002, 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029675862000000001, 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 656 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.106208+0000", 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.106208+0000", 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.106206+0000", 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.107132+0000", 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00092523400000000004 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.107148+0000", 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6798000000000001e-05 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.107159+0000", 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0708e-05 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.107217+0000", 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7839e-05 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.107294+0000", 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7256999999999996e-05 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.107396+0000", 2024-09-21T15:16:44.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101606 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.133093+0000", 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025696869000000001 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.135793+0000", 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027001310000000001 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.135855+0000", 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1607999999999997e-05 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.135884+0000", 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9057e-05 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:658 2.7 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.145176+0000", 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.45696901499999998, 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032792865999999997, 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 658 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145176+0000", 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145176+0000", 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145174+0000", 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145989+0000", 2024-09-21T15:16:44.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081450999999999995 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146020+0000", 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0716000000000002e-05 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146038+0000", 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8323000000000001e-05 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146109+0000", 2024-09-21T15:16:44.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1353999999999995e-05 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146200+0000", 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0729000000000005e-05 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146323+0000", 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012247899999999999 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.171871+0000", 2024-09-21T15:16:44.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025548455000000001 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177904+0000", 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060330699999999998 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177941+0000", 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6772e-05 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177969+0000", 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8337999999999999e-05 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:660 2.7 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.186802+0000", 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.41534337700000001, 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030017789999999999, 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 660 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.186802+0000", 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.186802+0000", 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.186800+0000", 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.187682+0000", 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088133500000000002 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.187709+0000", 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7477000000000001e-05 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.187726+0000", 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7231000000000002e-05 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.187808+0000", 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2185e-05 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.187887+0000", 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8139000000000005e-05 2024-09-21T15:16:44.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.187991+0000", 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104669 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.215775+0000", 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027784028999999998 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.216671+0000", 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089614000000000002 2024-09-21T15:16:44.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.216762+0000", 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0744999999999999e-05 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.216820+0000", 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7546000000000001e-05 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.705 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:662 2.7 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.225579+0000", 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.37656621899999998, 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027228094000000001, 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 662 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.225579+0000", 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.225579+0000", 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.225578+0000", 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226390+0000", 2024-09-21T15:16:44.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081214600000000003 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226404+0000", 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4076e-05 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226416+0000", 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2041e-05 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226469+0000", 2024-09-21T15:16:44.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2992999999999999e-05 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226604+0000", 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013453200000000001 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226761+0000", 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015785500000000001 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.250655+0000", 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023893544999999999 2024-09-21T15:16:44.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.252741+0000", 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020864360000000001 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.252779+0000", 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7732999999999999e-05 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.252807+0000", 2024-09-21T15:16:44.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8025999999999999e-05 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:664 2.7 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.259408+0000", 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.342737334, 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027871818, 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:44.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 664 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.259408+0000", 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.259408+0000", 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.259406+0000", 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260296+0000", 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088925500000000004 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260314+0000", 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8697000000000001e-05 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260328+0000", 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3628000000000001e-05 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260367+0000", 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9221000000000003e-05 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260437+0000", 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9904e-05 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260522+0000", 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4447e-05 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.286737+0000", 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026215171999999998 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.287214+0000", 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047761899999999998 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.287254+0000", 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9066000000000002e-05 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.287280+0000", 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6261000000000001e-05 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.716 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:44.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:44.718 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:44.712+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:44.819 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:44.817+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:44.872 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:44.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:693 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'54, mlcod=58'52)", 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.872521+0000", 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.002329509, 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023648649999999998, 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872521+0000", 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872521+0000", 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872519+0000", 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872615+0000", 2024-09-21T15:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6111000000000002e-05 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872644+0000", 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9343e-05 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872657+0000", 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2249000000000001e-05 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872714+0000", 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7111999999999998e-05 2024-09-21T15:16:44.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.872746+0000", 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2509000000000003e-05 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 1 2024-09-21T15:16:44.876 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:44.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:44.919 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:44.916+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:44.992 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:44.993 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:45.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:45.046 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:45.044+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:45.122 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:45.121+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:45.159 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:45.159 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.376357612000007, 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.160 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:45.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:624 2.7 e57/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 57'33, mlcod=57'31)", 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.737673526, 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029592388000000001, 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.166 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420904+0000", 2024-09-21T15:16:45.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421288+0000", 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000384674 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421318+0000", 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0017000000000001e-05 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421332+0000", 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421441+0000", 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109409 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421481+0000", 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0125e-05 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450452+0000", 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028971001 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450498+0000", 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5037999999999998e-05 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.69942039, 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:45.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:16:45.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:628 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'37, mlcod=57'35)", 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.6474827780000001, 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028343068999999999, 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511094+0000", 2024-09-21T15:16:45.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511478+0000", 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038358199999999999 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511509+0000", 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525999999999999e-05 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511521+0000", 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2223e-05 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511585+0000", 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4280000000000001e-05 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511618+0000", 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2795e-05 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539390+0000", 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02777191 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539439+0000", 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9230000000000001e-05 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:45.178 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.3714010210000001, 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.320239578, 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.184 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:45.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:646 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.2693409500000001, 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027542277, 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 646 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.190 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889236+0000", 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890378+0000", 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011416410000000001 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890406+0000", 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7756999999999999e-05 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890418+0000", 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890513+0000", 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4914999999999995e-05 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890608+0000", 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5076999999999998e-05 2024-09-21T15:16:45.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890723+0000", 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115009 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916147+0000", 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025424444000000001 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916701+0000", 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055418399999999999 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916743+0000", 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1640000000000001e-05 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916780+0000", 2024-09-21T15:16:45.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7092000000000003e-05 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.2321936120000001, 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:45.196 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.201 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:654 2.5 e57/56 2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head v 57'51, mlcod=57'49)", 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.064984+0000", 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.093594972, 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026648094000000001, 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064984+0000", 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064984+0000", 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.064982+0000", 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.065428+0000", 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044609400000000002 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.065456+0000", 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8362000000000001e-05 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.065467+0000", 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.118e-05 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.065551+0000", 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3372000000000001e-05 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.065585+0000", 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4054999999999998e-05 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.091543+0000", 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025958443000000001 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.091632+0000", 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8411000000000004e-05 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.205 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:658 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'43, mlcod=57'41)", 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.0121292150000001, 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031302035999999998, 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:45.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146448+0000", 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146818+0000", 2024-09-21T15:16:45.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036965199999999999 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146845+0000", 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7525000000000002e-05 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146858+0000", 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2703e-05 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146954+0000", 2024-09-21T15:16:45.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6186000000000004e-05 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146992+0000", 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177692+0000", 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030700439999999999 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177751+0000", 2024-09-21T15:16:45.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9271000000000002e-05 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:660 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'45, mlcod=57'43)", 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.188136+0000", 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.97044306199999997, 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025817343, 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188136+0000", 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188136+0000", 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188134+0000", 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188849+0000", 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00071548100000000002 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188888+0000", 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9082000000000003e-05 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188906+0000", 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7813999999999999e-05 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188963+0000", 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6410999999999999e-05 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.188982+0000", 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9592000000000001e-05 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.213858+0000", 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024875963000000001 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.213953+0000", 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4851000000000004e-05 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:662 2.7 e58/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 58'47, mlcod=57'45)", 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.226885+0000", 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.93169412500000004, 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025673959, 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.214 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226885+0000", 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226885+0000", 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.226883+0000", 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.227280+0000", 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039629500000000001 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.227298+0000", 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8297000000000001e-05 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.227306+0000", 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1429999999999998e-06 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.227363+0000", 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6530999999999998e-05 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.227387+0000", 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4689e-05 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.252513+0000", 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025125469000000001 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.252558+0000", 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5646000000000003e-05 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:664 2.7 e58/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 58'49, mlcod=58'47)", 2024-09-21T15:16:45.218 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.260620+0000", 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.89795900799999995, 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025931335999999999, 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260620+0000", 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260620+0000", 2024-09-21T15:16:45.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.260618+0000", 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.261045+0000", 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042693000000000001 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.261065+0000", 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.261073+0000", 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3280000000000006e-06 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.261128+0000", 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4873000000000001e-05 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.261158+0000", 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0343999999999999e-05 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.286498+0000", 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025340066000000001 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.286551+0000", 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2852000000000002e-05 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.222 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:674 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'7, mlcod=50'5)", 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.69306461500000005, 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030213265, 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:45.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465512+0000", 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465905+0000", 2024-09-21T15:16:45.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392972 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465940+0000", 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5026e-05 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465954+0000", 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3256e-05 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466019+0000", 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5701e-05 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466046+0000", 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6658000000000001e-05 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495689+0000", 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029642966999999999 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495727+0000", 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8352000000000001e-05 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.65166183300000002, 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.227 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:16:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:678 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'11, mlcod=58'9)", 2024-09-21T15:16:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.60331384700000001, 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026899573, 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555263+0000", 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555801+0000", 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000538085 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555824+0000", 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2161e-05 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555836+0000", 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2118e-05 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555885+0000", 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9515000000000003e-05 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555905+0000", 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9913999999999999e-05 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.582080+0000", 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026175006000000001 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.582164+0000", 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4208000000000004e-05 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:684 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.47021923700000001, 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028865808999999999, 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688358+0000", 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688856+0000", 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000498044 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688876+0000", 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.931e-05 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688886+0000", 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0814e-05 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688948+0000", 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1636000000000004e-05 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688980+0000", 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2020999999999997e-05 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717144+0000", 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028163898999999999 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717225+0000", 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1322000000000005e-05 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:688 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'61, mlcod=58'59)", 2024-09-21T15:16:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.390426259, 2024-09-21T15:16:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030671810000000001, 2024-09-21T15:16:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768151+0000", 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768527+0000", 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037648899999999998 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768553+0000", 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6327999999999999e-05 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768565+0000", 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1467999999999999e-05 2024-09-21T15:16:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768687+0000", 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121721 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768746+0000", 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200999999999998e-05 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798766+0000", 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030020013000000002 2024-09-21T15:16:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798824+0000", 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:690 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'51, mlcod=58'49)", 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.349758132, 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029415891, 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808819+0000", 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809316+0000", 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049685199999999999 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809333+0000", 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7133999999999999e-05 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809340+0000", 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9979999999999997e-06 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809443+0000", 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102642 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809479+0000", 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5539e-05 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838194+0000", 2024-09-21T15:16:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028715469 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838236+0000", 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2283000000000002e-05 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.245417151, 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:16:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:16:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.253 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:45.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:45.255 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:16:45.256 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:45.252+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:45.354 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:45.353+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:45.368 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:45.368 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:45.368 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:45.369 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:45.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:45.456 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:45.453+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:45.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:45.492 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:45.492 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:45.492 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:45.492 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:45.492 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:45.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:45.556 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:45.557+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:45.607 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:45.636 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:45.636 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:45.636 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 3.72641 s, 18.0 MB/s 2024-09-21T15:16:45.638 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:45.639 INFO:tasks.workunit.client.0.smithi146.stderr:++ dd if=/tmp/tmp.flUinkcHzA/data bs=1M 2024-09-21T15:16:45.639 INFO:tasks.workunit.client.0.smithi146.stderr:++ md5sum 2024-09-21T15:16:45.657 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:45.652+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:45.658 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:45.658 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:45.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.838757274000002, 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:45.663 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:45.663 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:45.664 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:45.669 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:45.686 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:45.708 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:45.729 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:45.758 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:45.757+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:45.776 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:45.776 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:45.776 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.136152 s, 493 MB/s 2024-09-21T15:16:45.778 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --no-progress export testrbdnbd17425 - 2024-09-21T15:16:45.778 INFO:tasks.workunit.client.0.smithi146.stderr:++ md5sum 2024-09-21T15:16:45.784 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:45.784 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:45.794 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:45.862 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:45.857+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:45.907 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:45.907 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:45.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:45.961 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:45.960+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:45.966 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:16:45.962+0000 7f2bd4795700 0 -- 172.21.15.146:0/906795322 >> [v2:172.21.15.142:6800/3327577487,v1:172.21.15.142:6801/3327577487] conn(0x7f2bb4008c10 msgr2=0x7f2bb400b0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:16:45.976 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:16:45.970+0000 7f2bd4795700 0 -- 172.21.15.146:0/906795322 >> [v2:172.21.15.142:6800/3327577487,v1:172.21.15.142:6801/3327577487] conn(0x7f2bb803ade0 msgr2=0x7f2bb803d1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:16:46.027 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.027 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:46.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' '2f0539b12ef79b6e31b3197ffd2bf2e4 -' = '2f0539b12ef79b6e31b3197ffd2bf2e4 -' ']' 2024-09-21T15:16:46.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 17820 2024-09-21T15:16:46.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:16:46.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=17820 2024-09-21T15:16:46.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:46.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:46.045 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:46.046 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:46.046 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:46.048 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:46.048 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:46.048 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:46.062 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:46.056+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:46.145 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.145 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:46.158 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:46.158 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^17820\b' 2024-09-21T15:16:46.159 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^17820\b' 2024-09-21T15:16:46.162 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:46.160+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:46.224 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:46.225 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:46.225 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *17820\b' 2024-09-21T15:16:46.225 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *17820\b' 2024-09-21T15:16:46.231 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:46.231 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:46.259 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.260 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.263 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:46.261+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:46.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:46.364 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:46.364+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:46.376 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.376 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:46.465 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:46.465+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:46.496 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.496 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:46.565 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:46.565+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:46.612 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.612 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:46.666 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:46.664+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:46.734 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd17425 2024-09-21T15:16:46.734 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:46.734 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:46.736 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:46.736 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:16:46.739 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:16:46.739 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:46.739 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd17425 2024-09-21T15:16:46.767 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:46.768+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:46.771 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.771 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:46.867 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:46.865+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:46.902 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:46.902 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:46.905 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:46.905 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:16:46.905 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:16:46.905 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:16:46.906 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:16:46.906 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:16:46.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:46.968 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:46.969+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:46.991 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=17932 2024-09-21T15:16:46.991 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 17932 2024-09-21T15:16:46.991 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 17932 -C rbd-nbd 2024-09-21T15:16:46.999 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:47.000 INFO:tasks.workunit.client.0.smithi146.stdout: 17932 ? 00:00:00 rbd-nbd 2024-09-21T15:16:47.000 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:47.000 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T15:16:47.021 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:47.021 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:47.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:47.069 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:47.069+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:47.070 INFO:tasks.workunit.client.0.smithi146.stderr:+ provisioned=67108864 2024-09-21T15:16:47.071 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:47.071 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T15:16:47.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:47.089 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:47.090 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T15:16:47.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:16:47.143 INFO:tasks.workunit.client.0.smithi146.stderr:+ used=67108864 2024-09-21T15:16:47.143 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-21T15:16:47.143 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-09-21T15:16:47.144 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-21T15:16:47.144 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:47.144 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:47.145 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:47.146 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:47.146 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:47.146 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:16:47.148 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:16:47.148 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:47.148 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-21T15:16:47.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:47.170 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:47.168+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:47.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:16:47.262 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:47.262 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:47.271 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:47.269+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:47.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:47.343 INFO:tasks.workunit.client.0.smithi146.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-09-21T15:16:47.344 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:47.344 INFO:tasks.workunit.client.0.smithi146.stderr:+ sync 2024-09-21T15:16:47.371 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:47.369+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 59, 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:16:45.196438+0000", 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:16:30.960731+0000", 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5801984, 2024-09-21T15:16:47.406 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-09-21T15:16:47.407 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "jewel", 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:16:47.408 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:16:47.409 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:16:47.410 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:16:47.411 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:16:47.412 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:16:47.413 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:16:47.414 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:16:47.415 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:16:47.416 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 20, 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:16:47.417 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:16:47.418 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:16:47.419 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.9900000095367432, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.9900000095367432, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.67000001668930054, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "new_default_pool17425", 2024-09-21T15:16:47.420 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:16:34.640599+0000", 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:16:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "54", 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:16:47.422 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 54, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:16:47.423 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:16:47.424 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.2400000095367432, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.2400000095367432, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.67000001668930054, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5, 2024-09-21T15:16:47.425 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:47.426 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 56, 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.429 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.430 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:47.431 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.427032470703125, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.432 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.433 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.434 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 0, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:47.437 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.438 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:47.440 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:16:47.441 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:47.442 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.443 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.444 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.445 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.446 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.447 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 0, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 17, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.449 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:16:47.450 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.451 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:47.452 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:16:47.453 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.455 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:16:47.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:16:47.457 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:47.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:47.459 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 65536, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:47.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 65536, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:16:47.461 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [], 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [], 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:16:47.462 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:16:47.463 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:16:47.464 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:16:47.464 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:16:47.464 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:16:47.464 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:16:47.464 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:16:47.464 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.464 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:47.466 INFO:tasks.thrashosds.thrasher:j is {'epoch': 59, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:16:45.196438+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:16:30.960731+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 3, 'pool_name': 'new_default_pool17425', 'create_time': '2024-09-21T15:16:34.640599+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.427032470703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:16:47.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:16:47.467 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:47.467 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:47.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:47.472 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:47.473+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:47.573 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:47.568+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:47.573 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:47.574 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:47.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:47.674 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:47.672+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:47.737 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:47.737 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:47.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:47.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:47.757 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:16:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:16:47.771 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-21T15:16:45.938383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134808242,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1243,"num_read_kb":68231,"num_write":277,"num_write_kb":133566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":389200,"kb_used_data":264720,"kb_used_omap":6,"kb_used_meta":124409,"kb_avail":377098160,"statfs":{"total":386547056640,"available":386148515840,"internally_reserved":0,"allocated":271073280,"data_stored":270520080,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":6373,"internal_metadata":127395611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001065"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":199,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.062036+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:16:38.062036+0000","last_peered":"2024-09-21T15:16:38.062036+0000","last_clean":"2024-09-21T15:16:38.062036+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:16:38.062036+0000","last_undegraded":"2024-09-21T15:16:38.062036+0000","last_fullsized":"2024-09-21T15:16:38.062036+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"50'5","reported_seq":225,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-21T15:16:41.317266+0000","last_change":"2024-09-21T15:16:32.457105+0000","last_active":"2024-09-21T15:16:41.317266+0000","last_peered":"2024-09-21T15:16:41.317266+0000","last_clean":"2024-09-21T15:16:41.317266+0000","last_became_active":"2024-09-21T15:16:32.449909+0000","last_became_peered":"2024-09-21T15:16:32.449909+0000","last_unstale":"2024-09-21T15:16:41.317266+0000","last_undegraded":"2024-09-21T15:16:41.317266+0000","last_fullsized":"2024-09-21T15:16:41.317266+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:37:06.923779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":145,"num_read_kb":9224,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"54'2","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.069669+0000","last_change":"2024-09-21T15:16:38.067320+0000","last_active":"2024-09-21T15:16:38.069669+0000","last_peered":"2024-09-21T15:16:38.069669+0000","last_clean":"2024-09-21T15:16:38.069669+0000","last_became_active":"2024-09-21T15:16:36.032402+0000","last_became_peered":"2024-09-21T15:16:36.032402+0000","last_unstale":"2024-09-21T15:16:38.069669+0000","last_undegraded":"2024-09-21T15:16:38.069669+0000","last_fullsized":"2024-09-21T15:16:38.069669+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:11:45.723062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181755,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"50'4","reported_seq":35,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.545937+0000","last_change":"2024-09-21T15:16:32.450144+0000","last_active":"2024-09-21T15:16:38.545937+0000","last_peered":"2024-09-21T15:16:38.545937+0000","last_clean":"2024-09-21T15:16:38.545937+0000","last_became_active":"2024-09-21T15:16:32.448296+0000","last_became_peered":"2024-09-21T15:16:32.448296+0000","last_unstale":"2024-09-21T15:16:38.545937+0000","last_undegraded":"2024-09-21T15:16:38.545937+0000","last_fullsized":"2024-09-21T15:16:38.545937+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:36.808016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388624,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.069740+0000","last_change":"2024-09-21T15:16:38.069740+0000","last_active":"2024-09-21T15:16:38.069740+0000","last_peered":"2024-09-21T15:16:38.069740+0000","last_clean":"2024-09-21T15:16:38.069740+0000","last_became_active":"2024-09-21T15:16:36.034793+0000","last_became_peered":"2024-09-21T15:16:36.034793+0000","last_unstale":"2024-09-21T15:16:38.069740+0000","last_undegraded":"2024-09-21T15:16:38.069740+0000","last_fullsized":"2024-09-21T15:16:38.069740+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:46.057197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066556199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"50'3","reported_seq":76,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-21T15:16:41.289310+0000","last_change":"2024-09-21T15:15:34.263162+0000","last_active":"2024-09-21T15:16:41.289310+0000","last_peered":"2024-09-21T15:16:41.289310+0000","last_clean":"2024-09-21T15:16:41.289310+0000","last_became_active":"2024-09-21T15:15:34.262327+0000","last_became_peered":"2024-09-21T15:15:34.262327+0000","last_unstale":"2024-09-21T15:16:41.289310+0000","last_undegraded":"2024-09-21T15:16:41.289310+0000","last_fullsized":"2024-09-21T15:16:41.289310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:15.380455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":4096,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.546425+0000","last_change":"2024-09-21T15:16:38.546425+0000","last_active":"2024-09-21T15:16:38.546425+0000","last_peered":"2024-09-21T15:16:38.546425+0000","last_clean":"2024-09-21T15:16:38.546425+0000","last_became_active":"2024-09-21T15:16:36.037559+0000","last_became_peered":"2024-09-21T15:16:36.037559+0000","last_unstale":"2024-09-21T15:16:38.546425+0000","last_undegraded":"2024-09-21T15:16:38.546425+0000","last_fullsized":"2024-09-21T15:16:38.546425+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:00:21.769849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046520300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"50'5","reported_seq":114,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-21T15:16:41.154478+0000","last_change":"2024-09-21T15:16:31.993192+0000","last_active":"2024-09-21T15:16:41.154478+0000","last_peered":"2024-09-21T15:16:41.154478+0000","last_clean":"2024-09-21T15:16:41.154478+0000","last_became_active":"2024-09-21T15:16:31.986939+0000","last_became_peered":"2024-09-21T15:16:31.986939+0000","last_unstale":"2024-09-21T15:16:41.154478+0000","last_undegraded":"2024-09-21T15:16:41.154478+0000","last_fullsized":"2024-09-21T15:16:41.154478+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:36.317852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":4112,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'1","reported_seq":17,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.067343+0000","last_change":"2024-09-21T15:16:38.067343+0000","last_active":"2024-09-21T15:16:38.067343+0000","last_peered":"2024-09-21T15:16:38.067343+0000","last_clean":"2024-09-21T15:16:38.067343+0000","last_became_active":"2024-09-21T15:16:36.036196+0000","last_became_peered":"2024-09-21T15:16:36.036196+0000","last_unstale":"2024-09-21T15:16:38.067343+0000","last_undegraded":"2024-09-21T15:16:38.067343+0000","last_fullsized":"2024-09-21T15:16:38.067343+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:00:58.292787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046455,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.070015+0000","last_change":"2024-09-21T15:16:38.070015+0000","last_active":"2024-09-21T15:16:38.070015+0000","last_peered":"2024-09-21T15:16:38.070015+0000","last_clean":"2024-09-21T15:16:38.070015+0000","last_became_active":"2024-09-21T15:16:36.034532+0000","last_became_peered":"2024-09-21T15:16:36.034532+0000","last_unstale":"2024-09-21T15:16:38.070015+0000","last_undegraded":"2024-09-21T15:16:38.070015+0000","last_fullsized":"2024-09-21T15:16:38.070015+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:16.910701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078075599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"50'8","reported_seq":89,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-21T15:16:41.126543+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:16:41.126543+0000","last_peered":"2024-09-21T15:16:41.126543+0000","last_clean":"2024-09-21T15:16:41.126543+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:16:41.126543+0000","last_undegraded":"2024-09-21T15:16:41.126543+0000","last_fullsized":"2024-09-21T15:16:41.126543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":25165824,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":35,"num_read_kb":4099,"num_write":8,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.546407+0000","last_change":"2024-09-21T15:16:38.546407+0000","last_active":"2024-09-21T15:16:38.546407+0000","last_peered":"2024-09-21T15:16:38.546407+0000","last_clean":"2024-09-21T15:16:38.546407+0000","last_became_active":"2024-09-21T15:16:36.035274+0000","last_became_peered":"2024-09-21T15:16:36.035274+0000","last_unstale":"2024-09-21T15:16:38.546407+0000","last_undegraded":"2024-09-21T15:16:38.546407+0000","last_fullsized":"2024-09-21T15:16:38.546407+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:16.310017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043867299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"50'25","reported_seq":292,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-21T15:16:41.180321+0000","last_change":"2024-09-21T15:16:31.988014+0000","last_active":"2024-09-21T15:16:41.180321+0000","last_peered":"2024-09-21T15:16:41.180321+0000","last_clean":"2024-09-21T15:16:41.180321+0000","last_became_active":"2024-09-21T15:16:31.987544+0000","last_became_peered":"2024-09-21T15:16:31.987544+0000","last_unstale":"2024-09-21T15:16:41.180321+0000","last_undegraded":"2024-09-21T15:16:41.180321+0000","last_fullsized":"2024-09-21T15:16:41.180321+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:55:44.330368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":20971574,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":316,"num_read_kb":13576,"num_write":41,"num_write_kb":20514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.063134+0000","last_change":"2024-09-21T15:16:38.063134+0000","last_active":"2024-09-21T15:16:38.063134+0000","last_peered":"2024-09-21T15:16:38.063134+0000","last_clean":"2024-09-21T15:16:38.063134+0000","last_became_active":"2024-09-21T15:16:36.031385+0000","last_became_peered":"2024-09-21T15:16:36.031385+0000","last_unstale":"2024-09-21T15:16:38.063134+0000","last_undegraded":"2024-09-21T15:16:38.063134+0000","last_fullsized":"2024-09-21T15:16:38.063134+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:15.662224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092366200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"55'26","reported_seq":247,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-21T15:16:41.263059+0000","last_change":"2024-09-21T15:16:32.449805+0000","last_active":"2024-09-21T15:16:41.263059+0000","last_peered":"2024-09-21T15:16:41.263059+0000","last_clean":"2024-09-21T15:16:41.263059+0000","last_became_active":"2024-09-21T15:16:32.449182+0000","last_became_peered":"2024-09-21T15:16:32.449182+0000","last_unstale":"2024-09-21T15:16:41.263059+0000","last_undegraded":"2024-09-21T15:16:41.263059+0000","last_fullsized":"2024-09-21T15:16:41.263059+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:50.916780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":317,"num_read_kb":16557,"num_write":12,"num_write_kb":16388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-21T15:16:38.070164+0000","last_change":"2024-09-21T15:16:38.070164+0000","last_active":"2024-09-21T15:16:38.070164+0000","last_peered":"2024-09-21T15:16:38.070164+0000","last_clean":"2024-09-21T15:16:38.070164+0000","last_became_active":"2024-09-21T15:16:36.034516+0000","last_became_peered":"2024-09-21T15:16:36.034516+0000","last_unstale":"2024-09-21T15:16:38.070164+0000","last_undegraded":"2024-09-21T15:16:38.070164+0000","last_fullsized":"2024-09-21T15:16:38.070164+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:23:49.580455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063897200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"50'24","reported_seq":204,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-21T15:16:41.207252+0000","last_change":"2024-09-21T15:15:24.830605+0000","last_active":"2024-09-21T15:16:41.207252+0000","last_peered":"2024-09-21T15:16:41.207252+0000","last_clean":"2024-09-21T15:16:41.207252+0000","last_became_active":"2024-09-21T15:15:22.827487+0000","last_became_peered":"2024-09-21T15:15:22.827487+0000","last_unstale":"2024-09-21T15:16:41.207252+0000","last_undegraded":"2024-09-21T15:16:41.207252+0000","last_fullsized":"2024-09-21T15:16:41.207252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:13.168371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":20971558,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":229,"num_read_kb":16466,"num_write":40,"num_write_kb":20513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134217855,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":1127,"num_read_kb":68131,"num_write":116,"num_write_kb":131148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268480512,"data_stored":268435729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76840,"kb_used_data":45720,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94295000,"statfs":{"total":96636764160,"available":96558080000,"internally_reserved":0,"allocated":46817280,"data_stored":46953627,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":1.0720000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.54200000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.85599999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.94399999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.998},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":1.028}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109556,"kb_used_data":78436,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94262284,"statfs":{"total":96636764160,"available":96524578816,"internally_reserved":0,"allocated":80318464,"data_stored":79917710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.96199999999999997}]},{"osd":1,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.82399999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.80400000000000005}]},{"osd":5,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.91600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101428,"kb_used_data":70308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270412,"statfs":{"total":96636764160,"available":96532901888,"internally_reserved":0,"allocated":71995392,"data_stored":72119546,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.76800000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.629},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.81100000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.84999999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.95999999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.0860000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101376,"kb_used_data":70256,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270464,"statfs":{"total":96636764160,"available":96532955136,"internally_reserved":0,"allocated":71942144,"data_stored":71529197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.31}]},{"osd":2,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.63},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.74099999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.70099999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.755}]}]},{"osd":4,"up_from":0,"seq":51539607575,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":51539607573,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71319552,"data_stored":71303279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71319552,"data_stored":71303279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79691792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":46141440,"data_stored":46137360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:16:47.772 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [4, 5] 2024-09-21T15:16:47.772 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.1', '4', '5'] 2024-09-21T15:16:47.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.1 4 5 2024-09-21T15:16:47.775 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:47.773+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:47.863 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:47.863 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:47.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:47.875 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:47.876+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:47.881 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:47.907 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:47.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.153056887999995, 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:47.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:47.977 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:47.978 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:47.978 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:47.977+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:47.983 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:48.002 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:48.003 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:48.080 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:48.077+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:48.097 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:48.097 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:48.097 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:48.098 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:48.108 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:48.181 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:48.180+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:48.210 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:48.210 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:48.210 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:48.210 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:48.210 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:48.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:16:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:set 2.1 pg_upmap mapping to [4,5] 2024-09-21T15:16:48.282 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:48.284+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:48.326 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.6995807430000003, 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.6591341509999999, 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:16:48.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.5834328060000002, 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.335 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.5368124090000004, 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:48.343 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.4861957849999996, 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:16:48.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.3978423129999999, 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.351 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:666 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'10, mlcod=50'8)", 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.303757+0000", 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.0205367030000003, 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026222367999999999, 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303757+0000", 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303757+0000", 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303756+0000", 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304224+0000", 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000468679 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304244+0000", 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304253+0000", 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9700000000000005e-06 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304330+0000", 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6402999999999993e-05 2024-09-21T15:16:48.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304379+0000", 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9777000000000001e-05 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.329942+0000", 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025562279 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.329979+0000", 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7879000000000001e-05 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:670 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'14, mlcod=58'12)", 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.381093+0000", 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.9432002939999999, 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025800970999999999, 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381093+0000", 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381093+0000", 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381092+0000", 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381373+0000", 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000280112 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381388+0000", 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5917e-05 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381395+0000", 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.866e-06 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381448+0000", 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2460000000000003e-05 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381475+0000", 2024-09-21T15:16:48.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.745e-05 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.406858+0000", 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025382978 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.406894+0000", 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6223999999999997e-05 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.9087114430000001, 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:16:48.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:16:48.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.363 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:700 2.6 e58/48 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head v 58'47, mlcod=58'45)", 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.998419+0000", 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.3258748809999998, 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026568365, 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.998419+0000", 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.998419+0000", 2024-09-21T15:16:48.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.998418+0000", 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999230+0000", 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081211499999999997 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999244+0000", 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4378e-05 2024-09-21T15:16:48.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999251+0000", 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4439999999999996e-06 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999294+0000", 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.32e-05 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999307+0000", 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2493e-05 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.024952+0000", 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025645364 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.024987+0000", 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5327999999999998e-05 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:704 2.6 e58/48 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head v 58'51, mlcod=58'49)", 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.077908+0000", 2024-09-21T15:16:48.367 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.2463856130000002, 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026283607, 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.077908+0000", 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.077908+0000", 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.077907+0000", 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078709+0000", 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00080141200000000004 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078722+0000", 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3687e-05 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078728+0000", 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4809999999999997e-06 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078771+0000", 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2901999999999997e-05 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078784+0000", 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3424000000000001e-05 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.104099+0000", 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025314789000000001 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.104192+0000", 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2829000000000002e-05 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.1938306010000002, 2024-09-21T15:16:48.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:16:48.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:16:48.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.1491237750000001, 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.375 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:16:48.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.1055163440000002, 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:16:48.379 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:16:48.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:16:48.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:712 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'59, mlcod=58'57)", 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.263456+0000", 2024-09-21T15:16:48.383 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.060837308, 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026044233999999999, 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.263456+0000", 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.263456+0000", 2024-09-21T15:16:48.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.263455+0000", 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264243+0000", 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000788133 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264273+0000", 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.025e-05 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264286+0000", 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2746e-05 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264327+0000", 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1409999999999998e-05 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264346+0000", 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8403e-05 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.289452+0000", 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025106590000000002 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.289501+0000", 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8415000000000003e-05 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.0092077669999999, 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:16:48.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:718 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'9, mlcod=58'7)", 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.392737+0000", 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.9315568110000001, 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025224995, 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.392 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392737+0000", 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392737+0000", 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392735+0000", 2024-09-21T15:16:48.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393115+0000", 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00037964800000000001 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393145+0000", 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9750000000000001e-05 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393157+0000", 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.22e-05 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393220+0000", 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2663999999999995e-05 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393240+0000", 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0789e-05 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.417928+0000", 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.024688009 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.417962+0000", 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3634999999999999e-05 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:48.396 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.848098368, 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:48.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:16:48.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.8027678979999999, 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:48.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 2.7235445149999999, 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.405 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:16:48.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:16:48.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:16:48.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.409 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:48.411 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:48.411 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:48.409+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:48.511 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:48.509+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:48.524 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:48.524 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:48.524 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:48.524 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:48.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:48.612 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:48.609+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:48.690 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:48.690 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:48.690 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:48.690 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:48.690 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:48.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:48.744 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:48.745+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 84.985361376, 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.804 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:48.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:48.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:48.808 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.975804792000005, 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:48.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:48.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:48.812 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.302583368000001, 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:48.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:48.816 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:48.817 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:48.818 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:48.816+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:48.918 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:48.920+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:48.926 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:48.926 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:48.926 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:48.926 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:48.935 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:49.019 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:49.020+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:49.043 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:49.043 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:49.043 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:49.043 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:49.043 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:49.052 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:49.120 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:49.117+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:49.165 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:49.165 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:49.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:49.165 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:49.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.165 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:49.165 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.166310425999995, 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:49.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.169 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:341 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0623637109999997, 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0082113480000000003, 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.170 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 341 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100526+0000", 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100530+0000", 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0460000000000001e-06 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100545+0000", 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5733999999999999e-05 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100552+0000", 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107817+0000", 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0072653520000000001 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107946+0000", 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012907000000000001 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.108738+0000", 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079230900000000005 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:343 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0603423920000008, 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081198239999999994, 2024-09-21T15:16:49.174 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 343 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:49.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102552+0000", 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6360000000000002e-06 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102584+0000", 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1813999999999997e-05 2024-09-21T15:16:49.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102593+0000", 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6430000000000001e-06 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.109769+0000", 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0071766089999999996 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.109886+0000", 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011657799999999999 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110668+0000", 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078233000000000003 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0602912149999995, 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:49.178 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0602401720000003, 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:49.182 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:16:49.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:16:49.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0446206290000006, 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.186 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:49.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:16:49.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:16:49.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0445938760000004, 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.190 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:16:49.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.8755457729999998, 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:16:49.194 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.200 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.8604009159999997, 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:16:49.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:16:49.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.8312540999999998, 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:16:49.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:16:49.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.8219271949999998, 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.211 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:16:49.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:16:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.7942910430000003, 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:16:49.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:16:49.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.221 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.7832133380000004, 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.7484437909999997, 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.227 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:16:49.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:16:49.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:16:49.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.232 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.8599197090000001, 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:16:49.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:16:49.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:16:49.237 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.849664642, 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.238 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:16:49.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:16:49.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:16:49.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.8176176449999999, 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:16:49.243 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:16:49.248 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.8054056919999999, 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:16:49.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:16:49.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:16:49.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:16:49.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.7715097339999999, 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:16:49.254 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.259 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.7324648649999999, 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:16:49.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:16:49.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:16:49.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.265 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:49.266 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:49.266 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:49.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:49.268 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:49.268+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:49.318 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:49.318 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T15:16:49.367 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:49.365+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:49.395 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:49.396 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:49.399 INFO:tasks.workunit.client.0.smithi146.stderr:+ provisioned=67108864 2024-09-21T15:16:49.400 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:49.400 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T15:16:49.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:49.468 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:49.469+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:49.478 INFO:tasks.workunit.client.0.smithi146.stderr:+ used=67108864 2024-09-21T15:16:49.478 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-21T15:16:49.478 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 17932 2024-09-21T15:16:49.478 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:16:49.478 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=17932 2024-09-21T15:16:49.478 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:49.478 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:49.479 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:49.480 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:49.480 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:49.483 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:49.483 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:49.483 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:49.512 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:49.512 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:49.512 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:49.512 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:49.512 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:49.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:49.569 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:49.568+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:49.587 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:49.587 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^17932\b' 2024-09-21T15:16:49.587 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^17932\b' 2024-09-21T15:16:49.628 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:49.628 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:49.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:49.628 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:49.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:622 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2536978550000004, 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033831148999999998, 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 622 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374818+0000", 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999886 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375659+0000", 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084095399999999999 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375676+0000", 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6812000000000001e-05 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375684+0000", 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0900000000000005e-06 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375731+0000", 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6737000000000003e-05 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375824+0000", 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3350000000000006e-05 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375959+0000", 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013498500000000001 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.407620+0000", 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031660559999999997 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408527+0000", 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090730699999999999 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408600+0000", 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3011999999999998e-05 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408661+0000", 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0785000000000002e-05 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:624 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.419667+0000", 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2088609610000001, 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031093571, 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 624 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.419667+0000", 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.419667+0000", 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.419665+0000", 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420490+0000", 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082458800000000001 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420509+0000", 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9584000000000001e-05 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420519+0000", 2024-09-21T15:16:49.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3030000000000005e-06 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420574+0000", 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4892999999999998e-05 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420656+0000", 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2313999999999998e-05 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420784+0000", 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127978 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.447966+0000", 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027181944999999999 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450694+0000", 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027282130000000002 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450732+0000", 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7799999999999997e-05 2024-09-21T15:16:49.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450760+0000", 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8479e-05 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.170717196, 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:632 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.580217+0000", 2024-09-21T15:16:49.644 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0483105869999996, 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032334979, 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 632 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.580217+0000", 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.580217+0000", 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.580216+0000", 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581059+0000", 2024-09-21T15:16:49.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084304599999999996 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581073+0000", 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4352000000000001e-05 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581079+0000", 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581158+0000", 2024-09-21T15:16:49.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8170999999999994e-05 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581310+0000", 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015192 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.581542+0000", 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023252900000000001 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.611597+0000", 2024-09-21T15:16:49.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030054953999999998 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.612488+0000", 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000891236 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.612528+0000", 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9579000000000002e-05 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.612552+0000", 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4216999999999999e-05 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:634 2.6 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.624869+0000", 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0036587780000001, 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033046593999999999, 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.650 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 634 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.624869+0000", 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.624869+0000", 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.624867+0000", 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625725+0000", 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085822800000000005 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625753+0000", 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7722999999999999e-05 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625766+0000", 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3436e-05 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625855+0000", 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8776000000000005e-05 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.625927+0000", 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1983999999999994e-05 2024-09-21T15:16:49.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.626023+0000", 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5970000000000004e-05 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.653662+0000", 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027639433000000001 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.657866+0000", 2024-09-21T15:16:49.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042037840000000003 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.657896+0000", 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9522999999999998e-05 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.657915+0000", 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.655 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:640 2.6 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.740998+0000", 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8875299930000002, 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033725333000000003, 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 640 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.740998+0000", 2024-09-21T15:16:49.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.740998+0000", 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.740996+0000", 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.741954+0000", 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095838900000000003 2024-09-21T15:16:49.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.741981+0000", 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6780000000000001e-05 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.741994+0000", 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3249e-05 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.742064+0000", 2024-09-21T15:16:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0019999999999997e-05 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.742110+0000", 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6014999999999999e-05 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.742180+0000", 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9437999999999995e-05 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.764290+0000", 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022110396000000001 2024-09-21T15:16:49.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.774674+0000", 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010383501999999999 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.774703+0000", 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9691000000000001e-05 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.774723+0000", 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9616e-05 2024-09-21T15:16:49.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:650 2.5 2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.983799+0000", 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6447288049999997, 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031828651999999999, 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.661 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 650 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.983799+0000", 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.983799+0000", 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.983797+0000", 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984706+0000", 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090949399999999995 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984762+0000", 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5365000000000003e-05 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984784+0000", 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1990000000000001e-05 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.984912+0000", 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012814400000000001 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.985060+0000", 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014847999999999999 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.985276+0000", 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021580899999999999 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.014713+0000", 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029436859999999999 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.015565+0000", 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085232400000000001 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.015602+0000", 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6522e-05 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.015627+0000", 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5758e-05 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:652 2.5 2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.022787+0000", 2024-09-21T15:16:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6057407440000002, 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033027978, 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 652 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.022787+0000", 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.022787+0000", 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.022785+0000", 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023706+0000", 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00092091600000000003 2024-09-21T15:16:49.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023737+0000", 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0749e-05 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023758+0000", 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1124e-05 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023843+0000", 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5828000000000006e-05 2024-09-21T15:16:49.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.023993+0000", 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014912700000000001 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.024186+0000", 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019344499999999999 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.054957+0000", 2024-09-21T15:16:49.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030771011000000001 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.055748+0000", 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079097 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.055778+0000", 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9859999999999999e-05 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.055815+0000", 2024-09-21T15:16:49.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7001e-05 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:658 2.7 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.145176+0000", 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4833513629999997, 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032792865999999997, 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.672 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 658 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145176+0000", 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145176+0000", 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145174+0000", 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.145989+0000", 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081450999999999995 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146020+0000", 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0716000000000002e-05 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146038+0000", 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8323000000000001e-05 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146109+0000", 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1353999999999995e-05 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146200+0000", 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0729000000000005e-05 2024-09-21T15:16:49.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146323+0000", 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012247899999999999 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.171871+0000", 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025548455000000001 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177904+0000", 2024-09-21T15:16:49.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060330699999999998 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177941+0000", 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6772e-05 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177969+0000", 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8337999999999999e-05 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:674 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.463841+0000", 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1646861550000001, 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032169672000000003, 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 674 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.463841+0000", 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.463841+0000", 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.463839+0000", 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.464755+0000", 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00091551299999999996 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.464787+0000", 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2342000000000003e-05 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.464801+0000", 2024-09-21T15:16:49.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3637e-05 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.464915+0000", 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114504 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465071+0000", 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155723 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465321+0000", 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000250385 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495592+0000", 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030270874 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495946+0000", 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035327900000000002 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495983+0000", 2024-09-21T15:16:49.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7426999999999999e-05 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.496011+0000", 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8078000000000001e-05 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:16:49.683 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1228329390000003, 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.0746610929999996, 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:16:49.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:16:49.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:16:49.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:688 2.5 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.766884+0000", 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8616437900000005, 2024-09-21T15:16:49.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032152053, 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 688 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.766884+0000", 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.766884+0000", 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.766882+0000", 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.767712+0000", 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083027899999999996 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.767742+0000", 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9342000000000001e-05 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.767759+0000", 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7643e-05 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.767841+0000", 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1934999999999994e-05 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.767927+0000", 2024-09-21T15:16:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6091e-05 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768037+0000", 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109771 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.795936+0000", 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027898648000000002 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798973+0000", 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0030371270000000001 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.799008+0000", 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5057e-05 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.799036+0000", 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.798e-05 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:690 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.807607+0000", 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8209211009999997, 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030922771000000002, 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 690 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.807607+0000", 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.807607+0000", 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.807605+0000", 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808435+0000", 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000829359 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808449+0000", 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4632e-05 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808457+0000", 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4839999999999998e-06 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808498+0000", 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.121e-05 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808577+0000", 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9102000000000003e-05 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808691+0000", 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113523 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.835771+0000", 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027080085 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838465+0000", 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026937699999999998 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838502+0000", 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7067999999999998e-05 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838529+0000", 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7716999999999999e-05 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7164919909999998, 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.705 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.5505081199999999, 2024-09-21T15:16:49.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:706 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.130182+0000", 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4983456630000003, 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033409731999999998, 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 706 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.130182+0000", 2024-09-21T15:16:49.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.130182+0000", 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.130163+0000", 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999809 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.130950+0000", 2024-09-21T15:16:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00078731899999999997 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.130981+0000", 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0923000000000002e-05 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.131000+0000", 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9355999999999999e-05 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.131097+0000", 2024-09-21T15:16:49.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6249000000000006e-05 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.131328+0000", 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023085799999999999 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.131672+0000", 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034476400000000001 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.159840+0000", 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028168147000000001 2024-09-21T15:16:49.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.163530+0000", 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036895790000000001 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.163563+0000", 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3013999999999999e-05 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.163592+0000", 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8685999999999999e-05 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:708 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.175396+0000", 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4531317479999997, 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030893271999999999, 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 708 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.721 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.175396+0000", 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.175396+0000", 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.175394+0000", 2024-09-21T15:16:49.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.176106+0000", 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00071270100000000005 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.176134+0000", 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7991e-05 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.176148+0000", 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3457e-05 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.176212+0000", 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4419999999999996e-05 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.176274+0000", 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1852000000000003e-05 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.176353+0000", 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8523999999999996e-05 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.201038+0000", 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024685528000000002 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.206077+0000", 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050392099999999997 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.206198+0000", 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120548 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.206289+0000", 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1161000000000004e-05 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:710 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.218840+0000", 2024-09-21T15:16:49.726 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4096878110000004, 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033517670999999999, 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 710 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.218840+0000", 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.218840+0000", 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.218838+0000", 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.219606+0000", 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076875900000000004 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.219626+0000", 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9819000000000001e-05 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.219642+0000", 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5548000000000001e-05 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.219718+0000", 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6639999999999998e-05 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.219848+0000", 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012925299999999999 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.220034+0000", 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018679100000000001 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.250056+0000", 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030021268 2024-09-21T15:16:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.252260+0000", 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002204461 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.252317+0000", 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6944999999999998e-05 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.252357+0000", 2024-09-21T15:16:49.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0429000000000002e-05 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3650028509999999, 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:16:49.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:16:49.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:16:49.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:49.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:49.737 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:49.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:49.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:49.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *17932\b' 2024-09-21T15:16:49.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *17932\b' 2024-09-21T15:16:49.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:49.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:49.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:49.740 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:49.740+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:49.840 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:49.840+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:49.895 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:49.895 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:49.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:49.941 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:49.941+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:50.014 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:50.014 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:50.014 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:50.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:50.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:50.041 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:50.041+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:50.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.347448400000005, 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:50.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:624 2.7 e57/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 57'33, mlcod=57'31)", 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.7087643139999997, 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029592388000000001, 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420904+0000", 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421288+0000", 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000384674 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421318+0000", 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0017000000000001e-05 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421332+0000", 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421441+0000", 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109409 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421481+0000", 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0125e-05 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450452+0000", 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028971001 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450498+0000", 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5037999999999998e-05 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.6705111779999999, 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:16:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:16:50.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:16:50.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:16:50.145 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.145 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:628 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'37, mlcod=57'35)", 2024-09-21T15:16:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.6185735660000002, 2024-09-21T15:16:50.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028343068999999999, 2024-09-21T15:16:50.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.147 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511094+0000", 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511478+0000", 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038358199999999999 2024-09-21T15:16:50.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511509+0000", 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525999999999999e-05 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511521+0000", 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2223e-05 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511585+0000", 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4280000000000001e-05 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511618+0000", 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2795e-05 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539390+0000", 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02777191 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539439+0000", 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9230000000000001e-05 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.3424918090000002, 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:16:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2913303660000004, 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:16:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:16:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:646 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2404317379999998, 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027542277, 2024-09-21T15:16:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 646 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889236+0000", 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890378+0000", 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011416410000000001 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890406+0000", 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7756999999999999e-05 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890418+0000", 2024-09-21T15:16:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890513+0000", 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4914999999999995e-05 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890608+0000", 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5076999999999998e-05 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890723+0000", 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115009 2024-09-21T15:16:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916147+0000", 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025424444000000001 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916701+0000", 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055418399999999999 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916743+0000", 2024-09-21T15:16:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1640000000000001e-05 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916780+0000", 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7092000000000003e-05 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2032844000000003, 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:16:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:16:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:16:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:16:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:658 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'43, mlcod=57'41)", 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.9832200029999996, 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031302035999999998, 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146448+0000", 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146818+0000", 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036965199999999999 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146845+0000", 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7525000000000002e-05 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146858+0000", 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2703e-05 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146954+0000", 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6186000000000004e-05 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146992+0000", 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177692+0000", 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030700439999999999 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177751+0000", 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9271000000000002e-05 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:674 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'7, mlcod=50'5)", 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6641554029999996, 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030213265, 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465512+0000", 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465905+0000", 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392972 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465940+0000", 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5026e-05 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465954+0000", 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3256e-05 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466019+0000", 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5701e-05 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466046+0000", 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6658000000000001e-05 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495689+0000", 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029642966999999999 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495727+0000", 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8352000000000001e-05 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6227526210000001, 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:16:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:16:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:16:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:678 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'11, mlcod=58'9)", 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5744046350000005, 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026899573, 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555263+0000", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555801+0000", 2024-09-21T15:16:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000538085 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555824+0000", 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2161e-05 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555836+0000", 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2118e-05 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555885+0000", 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9515000000000003e-05 2024-09-21T15:16:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555905+0000", 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9913999999999999e-05 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.582080+0000", 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026175006000000001 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.582164+0000", 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4208000000000004e-05 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:684 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4413100249999999, 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028865808999999999, 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688358+0000", 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688856+0000", 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000498044 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688876+0000", 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.931e-05 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688886+0000", 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0814e-05 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688948+0000", 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1636000000000004e-05 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688980+0000", 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2020999999999997e-05 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717144+0000", 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028163898999999999 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717225+0000", 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1322000000000005e-05 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:688 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'61, mlcod=58'59)", 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.3615170469999995, 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030671810000000001, 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768151+0000", 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768527+0000", 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037648899999999998 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768553+0000", 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6327999999999999e-05 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768565+0000", 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1467999999999999e-05 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768687+0000", 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121721 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768746+0000", 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200999999999998e-05 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798766+0000", 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030020013000000002 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798824+0000", 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:690 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'51, mlcod=58'49)", 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.3208489199999995, 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029415891, 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808819+0000", 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809316+0000", 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049685199999999999 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809333+0000", 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7133999999999999e-05 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809340+0000", 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9979999999999997e-06 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809443+0000", 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102642 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809479+0000", 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5539e-05 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838194+0000", 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028715469 2024-09-21T15:16:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838236+0000", 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2283000000000002e-05 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.2165079390000004, 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:16:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:16:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.6537241690000002, 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:725 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.6081020729999995, 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029546688000000002, 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 725 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521565+0000", 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:16:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081585200000000005 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522413+0000", 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2228999999999999e-05 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522431+0000", 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522491+0000", 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9710000000000003e-05 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522591+0000", 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010077300000000001 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522752+0000", 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000160963 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.545613+0000", 2024-09-21T15:16:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02286036 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551049+0000", 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005436771 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551086+0000", 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6477999999999999e-05 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551114+0000", 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8136e-05 2024-09-21T15:16:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.571015396, 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:16:50.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:16:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:16:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.5288961219999999, 2024-09-21T15:16:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:16:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:50.234 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:50.235 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:50.233+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:50.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:50.236 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:16:50.236 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:50.237 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:50.237 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:50.237 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:16:50.237 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:16:50.237 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:50.237 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:16:50.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:50.237 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:50.238 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:50.238 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:50.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:16:50.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:50.304 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:50.306 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:50.335 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:50.333+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:50.339 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:50.339 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:16:50.339 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:16:50.339 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:16:50.340 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:16:50.340 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:16:50.360 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:50.360 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:50.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:50.424 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18090 2024-09-21T15:16:50.424 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18090 2024-09-21T15:16:50.425 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18090 -C rbd-nbd 2024-09-21T15:16:50.436 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:50.436 INFO:tasks.workunit.client.0.smithi146.stdout: 18090 ? 00:00:00 rbd-nbd 2024-09-21T15:16:50.436 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:50.433+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:50.437 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:50.437 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T15:16:50.504 INFO:tasks.workunit.client.0.smithi146.stderr:+ provisioned=67108864 2024-09-21T15:16:50.505 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:50.505 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T15:16:50.525 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:50.525 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:50.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:50.536 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:50.537+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ used=67108864 2024-09-21T15:16:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-21T15:16:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-21T15:16:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:50.578 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:50.580 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:50.581 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:16:50.583 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:16:50.583 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:50.583 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-21T15:16:50.636 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:50.636+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:50.652 INFO:tasks.workunit.client.0.smithi146.stderr:+ sync 2024-09-21T15:16:50.656 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:50.656 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:50.666 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:50.738 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:50.740+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:50.780 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:50.780 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:50.790 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:50.819 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:50.819 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T15:16:50.838 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:50.840+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:50.895 INFO:tasks.workunit.client.0.smithi146.stderr:+ provisioned=67108864 2024-09-21T15:16:50.896 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd -p rbd --format xml du testrbdnbd17425 2024-09-21T15:16:50.896 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T15:16:50.903 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:50.903 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:50.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:50.939 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:50.940+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ used=0 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 0 -lt 67108864 ']' 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 18090 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18090 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:50.970 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:50.972 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:50.972 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:50.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:50.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:50.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:51.017 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.017 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.027 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:51.040 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:51.040+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:51.083 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18090\b' 2024-09-21T15:16:51.083 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18090\b' 2024-09-21T15:16:51.083 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:51.131 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.131 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.141 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:51.142 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:51.140+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:51.152 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:51.153 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:51.153 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18090\b' 2024-09-21T15:16:51.153 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18090\b' 2024-09-21T15:16:51.165 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:51.165 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:51.241 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:51.240+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:51.245 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.245 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.251 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:16:51.256 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:51.342 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:51.341+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:51.357 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.357 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:51.443 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:51.444+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:51.475 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.475 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.485 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:51.544 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:51.541+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:51.594 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.594 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.605 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:51.644 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:51.644+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:51.667 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd17425 2024-09-21T15:16:51.667 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:51.668 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:51.670 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:51.670 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:16:51.673 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:16:51.673 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:51.673 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd17425 2024-09-21T15:16:51.745 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:51.744+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:51.756 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.756 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:51.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:51.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:16:51.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:16:51.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:16:51.843 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:16:51.843 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:16:51.846 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:51.848+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:51.888 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:51.888 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:51.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:51.932 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18248 2024-09-21T15:16:51.932 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18248 2024-09-21T15:16:51.932 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18248 -C rbd-nbd 2024-09-21T15:16:51.945 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:51.945 INFO:tasks.workunit.client.0.smithi146.stdout: 18248 ? 00:00:00 rbd-nbd 2024-09-21T15:16:51.945 INFO:tasks.workunit.client.0.smithi146.stderr:++ basename /dev/nbd0 2024-09-21T15:16:51.946 INFO:tasks.workunit.client.0.smithi146.stderr:+ devname=nbd0 2024-09-21T15:16:51.947 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-21T15:16:51.947 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:51.945+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:51.950 INFO:tasks.workunit.client.0.smithi146.stderr:+ blocks=65536 2024-09-21T15:16:51.950 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 65536 2024-09-21T15:16:51.950 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd resize rbd/testrbdnbd17425 --size 128M 2024-09-21T15:16:52.011 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:52.011 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:52.018 INFO:tasks.workunit.client.0.smithi146.stderr: Resizing image: 100% complete...done. 2024-09-21T15:16:52.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:52.034 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd info rbd/testrbdnbd17425 2024-09-21T15:16:52.066 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:52.068+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout:rbd image 'testrbdnbd17425': 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: size 128 MiB in 32 objects 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: order 22 (4 MiB objects) 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: snapshot_count: 0 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: id: 13743413a48b 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: block_name_prefix: rbd_data.13743413a48b 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: format: 2 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: op_features: 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: flags: 2024-09-21T15:16:52.095 INFO:tasks.workunit.client.0.smithi146.stdout: create_timestamp: Sat Sep 21 15:16:32 2024 2024-09-21T15:16:52.096 INFO:tasks.workunit.client.0.smithi146.stdout: access_timestamp: Sat Sep 21 15:16:32 2024 2024-09-21T15:16:52.096 INFO:tasks.workunit.client.0.smithi146.stdout: modify_timestamp: Sat Sep 21 15:16:32 2024 2024-09-21T15:16:52.103 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-21T15:16:52.106 INFO:tasks.workunit.client.0.smithi146.stderr:+ blocks2=131072 2024-09-21T15:16:52.107 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 131072 2024-09-21T15:16:52.107 INFO:tasks.workunit.client.0.smithi146.stderr:+ test 131072 -eq 131072 2024-09-21T15:16:52.107 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd resize rbd/testrbdnbd17425 --allow-shrink --size 64M 2024-09-21T15:16:52.123 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:52.123 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:52.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:52.148 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:52.148+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:52.181 INFO:tasks.workunit.client.0.smithi146.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-09-21T15:16:52.198 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ blocks2=65536 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 65536 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ test 65536 -eq 65536 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 18248 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18248 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:52.203 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:52.204 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:52.204 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:52.206 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:52.206 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:52.206 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:52.243 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:52.243 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:52.248 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:52.249+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:52.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:52.303 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:52.303 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18248\b' 2024-09-21T15:16:52.304 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18248\b' 2024-09-21T15:16:52.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:52.349 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:52.348+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:52.371 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:52.372 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:52.372 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18248\b' 2024-09-21T15:16:52.372 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18248\b' 2024-09-21T15:16:52.383 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:52.383 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:52.419 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:52.419 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:52.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:52.469 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:52.470 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:52.468+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:52.493 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:52.530 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:52.550 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:52.552+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:52.582 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:52.585 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 94.802280968000005, 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.586 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:52.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:624 2.7 e57/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 57'33, mlcod=57'31)", 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1635968820000002, 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029592388000000001, 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420904+0000", 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421288+0000", 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000384674 2024-09-21T15:16:52.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421318+0000", 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0017000000000001e-05 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421332+0000", 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421441+0000", 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109409 2024-09-21T15:16:52.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421481+0000", 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0125e-05 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450452+0000", 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028971001 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450498+0000", 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5037999999999998e-05 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.595 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1253437460000004, 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:16:52.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:628 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'37, mlcod=57'35)", 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.0734061340000007, 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028343068999999999, 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511096+0000", 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511094+0000", 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511478+0000", 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038358199999999999 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511509+0000", 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0525999999999999e-05 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511521+0000", 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2223e-05 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511585+0000", 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4280000000000001e-05 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.511618+0000", 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2795e-05 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539390+0000", 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02777191 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.539439+0000", 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9230000000000001e-05 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7973243770000007, 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:52.609 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7461629340000009, 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:16:52.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.614 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:646 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.6952643060000003, 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027542277, 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 646 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.615 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889238+0000", 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.889236+0000", 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890378+0000", 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011416410000000001 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890406+0000", 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7756999999999999e-05 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890418+0000", 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890513+0000", 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4914999999999995e-05 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890608+0000", 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5076999999999998e-05 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.890723+0000", 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115009 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916147+0000", 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025424444000000001 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916701+0000", 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055418399999999999 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916743+0000", 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1640000000000001e-05 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.916780+0000", 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7092000000000003e-05 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.6581169679999999, 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.620 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:658 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'43, mlcod=57'41)", 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.4380525710000001, 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031302035999999998, 2024-09-21T15:16:52.625 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146448+0000", 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146818+0000", 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036965199999999999 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146845+0000", 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7525000000000002e-05 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146858+0000", 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2703e-05 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146954+0000", 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6186000000000004e-05 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146992+0000", 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177692+0000", 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030700439999999999 2024-09-21T15:16:52.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177751+0000", 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9271000000000002e-05 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:674 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'7, mlcod=50'5)", 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.1189879709999992, 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030213265, 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.629 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465512+0000", 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465905+0000", 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392972 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465940+0000", 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5026e-05 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465954+0000", 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3256e-05 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466019+0000", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5701e-05 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466046+0000", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6658000000000001e-05 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495689+0000", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029642966999999999 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495727+0000", 2024-09-21T15:16:52.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8352000000000001e-05 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.0775851890000006, 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:16:52.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:678 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'11, mlcod=58'9)", 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.0292372029999992, 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026899573, 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555265+0000", 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555263+0000", 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555801+0000", 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000538085 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555824+0000", 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2161e-05 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555836+0000", 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2118e-05 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555885+0000", 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9515000000000003e-05 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555905+0000", 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9913999999999999e-05 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.582080+0000", 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026175006000000001 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.582164+0000", 2024-09-21T15:16:52.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4208000000000004e-05 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:684 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8961425930000004, 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028865808999999999, 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:52.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688358+0000", 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688856+0000", 2024-09-21T15:16:52.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000498044 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688876+0000", 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.931e-05 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688886+0000", 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0814e-05 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688948+0000", 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1636000000000004e-05 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688980+0000", 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2020999999999997e-05 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717144+0000", 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028163898999999999 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717225+0000", 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1322000000000005e-05 2024-09-21T15:16:52.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:688 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'61, mlcod=58'59)", 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.816349615, 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030671810000000001, 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:52.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768151+0000", 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768527+0000", 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037648899999999998 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768553+0000", 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6327999999999999e-05 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768565+0000", 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1467999999999999e-05 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768687+0000", 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121721 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768746+0000", 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200999999999998e-05 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798766+0000", 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030020013000000002 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798824+0000", 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:690 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'51, mlcod=58'49)", 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.775681488, 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029415891, 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808819+0000", 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809316+0000", 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049685199999999999 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809333+0000", 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7133999999999999e-05 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809340+0000", 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9979999999999997e-06 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809443+0000", 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102642 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809479+0000", 2024-09-21T15:16:52.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5539e-05 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838194+0000", 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028715469 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838236+0000", 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2283000000000002e-05 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.653 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.671340507, 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:52.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:16:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.1085567369999998, 2024-09-21T15:16:52.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:16:52.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:725 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.062934641, 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029546688000000002, 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 725 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521565+0000", 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081585200000000005 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522413+0000", 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2228999999999999e-05 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522431+0000", 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522491+0000", 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9710000000000003e-05 2024-09-21T15:16:52.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522591+0000", 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010077300000000001 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522752+0000", 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000160963 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.545613+0000", 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02286036 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551049+0000", 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005436771 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551086+0000", 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6477999999999999e-05 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551114+0000", 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8136e-05 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.0258479640000004, 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.9837286899999995, 2024-09-21T15:16:52.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:16:52.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:16:52.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:16:52.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:52.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:52.679 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:52.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:52.680 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:52.681 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:52.682 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:52.681+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:52.783 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:52.784+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:52.796 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:52.796 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:52.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:52.885 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:52.884+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:52.885 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd17425 2024-09-21T15:16:52.886 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:52.886 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:52.888 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:52.888 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:16:52.891 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:16:52.891 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:52.891 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd17425 2024-09-21T15:16:52.959 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:52.959 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:52.959 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:52.959 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:52.959 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:52.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:52.985 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:52.985+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:53.054 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd list 2024-09-21T15:16:53.056 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk -v pool=rbd -v img=testrbdnbd17425 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-09-21T15:16:53.079 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.259425336999996, 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:53.083 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:53.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:53.088+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:53.089 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18376 2024-09-21T15:16:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18376 2024-09-21T15:16:53.136 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18376 -C rbd-nbd 2024-09-21T15:16:53.146 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:53.146 INFO:tasks.workunit.client.0.smithi146.stdout: 18376 ? 00:00:00 rbd-nbd 2024-09-21T15:16:53.146 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-21T15:16:53.146 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:53.147 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:53.149 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:53.149 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:16:53.150 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:16:53.150 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:53.150 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-21T15:16:53.186 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:53.188+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:53.208 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:53.208 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:53.208 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:53.208 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:53.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:53.254 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:53.254 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:53.255 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-21T15:16:53.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-09-21T15:16:53.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:16:53.266 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:53.266 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:53.266 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.109795 s, 611 MB/s 2024-09-21T15:16:53.267 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T15:16:53.267 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T15:16:53.267 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:53.268 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:53.269 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:53.270 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:16:53.272 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:16:53.272 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:53.272 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T15:16:53.282 INFO:tasks.workunit.client.0.smithi146.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-09-21T15:16:53.282 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:16:53.282 INFO:tasks.workunit.client.0.smithi146.stderr:0+0 records out 2024-09-21T15:16:53.282 INFO:tasks.workunit.client.0.smithi146.stderr:0 bytes copied, 0.00111263 s, 0.0 kB/s 2024-09-21T15:16:53.284 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:53.284 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 18376 2024-09-21T15:16:53.284 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:16:53.284 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18376 2024-09-21T15:16:53.284 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:53.284 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:53.284 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:53.285 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:53.286 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:53.289 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:53.289 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:53.289 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:53.298 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:53.300+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:53.324 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:53.325 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:53.325 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:53.325 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:53.325 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:53.335 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:53.385 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:53.385 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18376\b' 2024-09-21T15:16:53.385 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18376\b' 2024-09-21T15:16:53.387 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:53.385+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:53.443 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:53.444 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.8173704530000006, 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:53.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.448 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.7769238610000002, 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:16:53.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:16:53.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.7012225159999996, 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:53.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:16:53.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:16:53.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.6546021189999998, 2024-09-21T15:16:53.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:16:53.461 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.6039854949999999, 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:16:53.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:16:53.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.5156320230000002, 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:16:53.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:666 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'10, mlcod=50'8)", 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.303757+0000", 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.1383264129999997, 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026222367999999999, 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303757+0000", 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303757+0000", 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303756+0000", 2024-09-21T15:16:53.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304224+0000", 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000468679 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304244+0000", 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304253+0000", 2024-09-21T15:16:53.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9700000000000005e-06 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304330+0000", 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6402999999999993e-05 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.304379+0000", 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9777000000000001e-05 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.329942+0000", 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025562279 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.329979+0000", 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7879000000000001e-05 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.474 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:670 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'14, mlcod=58'12)", 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.381093+0000", 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.0609900040000007, 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025800970999999999, 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381093+0000", 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381093+0000", 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381092+0000", 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381373+0000", 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000280112 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381388+0000", 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5917e-05 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381395+0000", 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.866e-06 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381448+0000", 2024-09-21T15:16:53.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2460000000000003e-05 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.381475+0000", 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.745e-05 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.406858+0000", 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025382978 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.406894+0000", 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6223999999999997e-05 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.0265011529999999, 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.479 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:16:53.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:16:53.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:700 2.6 e58/48 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head v 58'47, mlcod=58'45)", 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.998419+0000", 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.4436645909999992, 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026568365, 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.998419+0000", 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.998419+0000", 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.998418+0000", 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999230+0000", 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081211499999999997 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999244+0000", 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4378e-05 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999251+0000", 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4439999999999996e-06 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999294+0000", 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.32e-05 2024-09-21T15:16:53.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.999307+0000", 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2493e-05 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.024952+0000", 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025645364 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.024987+0000", 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5327999999999998e-05 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:704 2.6 e58/48 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head v 58'51, mlcod=58'49)", 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.077908+0000", 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.3641753229999996, 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026283607, 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.488 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.077908+0000", 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.077908+0000", 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.077907+0000", 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078709+0000", 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00080141200000000004 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078722+0000", 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3687e-05 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078728+0000", 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4809999999999997e-06 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078771+0000", 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2901999999999997e-05 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.078784+0000", 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3424000000000001e-05 2024-09-21T15:16:53.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.104099+0000", 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025314789000000001 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.104192+0000", 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2829000000000002e-05 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.3116203110000004, 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:16:53.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.2669134849999999, 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.497 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:16:53.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:16:53.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.223306054, 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:16:53.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:712 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'59, mlcod=58'57)", 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.263456+0000", 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.1786270180000002, 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026044233999999999, 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.506 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.263456+0000", 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.263456+0000", 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.263455+0000", 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264243+0000", 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000788133 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264273+0000", 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.025e-05 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264286+0000", 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2746e-05 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264327+0000", 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1409999999999998e-05 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.264346+0000", 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8403e-05 2024-09-21T15:16:53.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.289452+0000", 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025106590000000002 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.289501+0000", 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8415000000000003e-05 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.1269974769999997, 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:16:53.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:718 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'9, mlcod=58'7)", 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.392737+0000", 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0493465210000004, 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025224995, 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.515 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392737+0000", 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392737+0000", 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392735+0000", 2024-09-21T15:16:53.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393115+0000", 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00037964800000000001 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393145+0000", 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9750000000000001e-05 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393157+0000", 2024-09-21T15:16:53.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.22e-05 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393220+0000", 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2663999999999995e-05 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.393240+0000", 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0789e-05 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.417928+0000", 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.024688009 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.417962+0000", 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3634999999999999e-05 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:53.519 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.9658880779999999, 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.523 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.9205576080000002, 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:16:53.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.8413342249999998, 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:16:53.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.532 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:53.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:53.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18376\b' 2024-09-21T15:16:53.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18376\b' 2024-09-21T15:16:53.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:53.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:53.535 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:53.535 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:53.532+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:53.566 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-09-21T15:16:53.568+0000 7f6b15ef9700 -1 mon.a@0(leader).osd e61 definitely_dead 0 2024-09-21T15:16:53.634 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:53.633+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:53.649 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:53.650 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:53.650 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:53.660 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:53.735 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:53.733+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:53.767 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:53.767 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:53.767 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:53.767 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:53.767 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:53.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:53.836 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:53.836+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:53.933 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:53.933 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:53.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:53.933 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:53.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.114499742999996, 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:53.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:53.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:53.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 89.104943159000001, 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.938 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:53.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.942 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.431721734999996, 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:53.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:53.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:53.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:53.946 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:53.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:53.948 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:53.945+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:53.959 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd17425 2024-09-21T15:16:53.959 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:53.959 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:53.961 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:53.962 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:16:53.965 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:16:53.965 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:53.965 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd17425 2024-09-21T15:16:54.048 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:54.048+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:54.053 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:54.053 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:54.053 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:54.053 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:54.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:54.135 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:54.135 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:16:54.135 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:16:54.135 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:16:54.136 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:16:54.136 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:16:54.148 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:54.148+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:54.173 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:54.173 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:54.173 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:54.173 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:54.173 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:54.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:54.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18450 2024-09-21T15:16:54.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18450 2024-09-21T15:16:54.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18450 -C rbd-nbd 2024-09-21T15:16:54.222 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:54.222 INFO:tasks.workunit.client.0.smithi146.stdout: 18450 ? 00:00:00 rbd-nbd 2024-09-21T15:16:54.222 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T15:16:54.222 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:54.223 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:54.225 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:54.225 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:16:54.227 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:16:54.227 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:54.227 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T15:16:54.249 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:54.248+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:54.267 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:16:54.277 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.2. 2024-09-21T15:16:54.290 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-21T15:16:54.298 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:54.298 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4198.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'43, mlcod=15'39)", 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.300022412999994, 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081374900000000007, 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.299 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996580+0000", 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996636+0000", 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6789000000000001e-05 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996656+0000", 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9496000000000001e-05 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.996664+0000", 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3960000000000006e-06 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999082+0000", 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0024174840000000001 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:17.999131+0000", 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8724999999999999e-05 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004646+0000", 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055155450000000002 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:18.004718+0000", 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1804999999999994e-05 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:341 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.196075698, 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0082113480000000003, 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 341 2024-09-21T15:16:54.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100527+0000", 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100526+0000", 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100530+0000", 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0460000000000001e-06 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100545+0000", 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5733999999999999e-05 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.100552+0000", 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107817+0000", 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0072653520000000001 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.107946+0000", 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012907000000000001 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.108738+0000", 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079230900000000005 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:343 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.194054379000001, 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081198239999999994, 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:54.308 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 343 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102548+0000", 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102552+0000", 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6360000000000002e-06 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102584+0000", 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1813999999999997e-05 2024-09-21T15:16:54.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102593+0000", 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6430000000000001e-06 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.109769+0000", 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0071766089999999996 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.109886+0000", 2024-09-21T15:16:54.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011657799999999999 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110668+0000", 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078233000000000003 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.194003201999999, 2024-09-21T15:16:54.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:54.316 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.193952159, 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:16:54.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:16:54.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.178332616, 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.178305863, 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.329 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.009257760000001, 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:54.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.9941129029999995, 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:16:54.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.9649660870000005, 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.341 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:16:54.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:16:54.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:54.346 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.9556391820000005, 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.351 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.9280030299999993, 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.352 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:16:54.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.357 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.9169253249999993, 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:16:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.8821557779999996, 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:16:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9936316959999996, 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:16:54.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:16:54.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:16:54.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9833766290000003, 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:16:54.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9513296320000002, 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.381 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:16:54.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9391176790000006, 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.387 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:16:54.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:16:54.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9052217210000002, 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.393 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:54.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:16:54.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:16:54.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.8661768520000006, 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:16:54.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:16:54.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:16:54.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:16:54.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:16:54.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:54.404 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:54.406 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:54.405+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:54.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:54.507 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:54.508+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:47 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 2097152~131072 in=131072b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.519495+0000", 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0073102109999999996, 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0073734020000000003, 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-09-21T15:16:54.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 47 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519495+0000", 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519495+0000", 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519488+0000", 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999928 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519570+0000", 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1225999999999998e-05 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519600+0000", 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9955e-05 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519618+0000", 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8797e-05 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519716+0000", 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7980999999999996e-05 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.519877+0000", 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016036300000000001 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.520090+0000", 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021370199999999999 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.529 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.5206.0:47 2.6 e62/48)", 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.525086+0000", 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.001719803, 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019871540000000001, 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "queued for pg", 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.525086+0000", 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.525086+0000", 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.525078+0000", 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999919 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.525096+0000", 2024-09-21T15:16:54.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8768999999999999e-05 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.525128+0000", 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1155999999999998e-05 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.525144+0000", 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6685999999999999e-05 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:16:54.532 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 2 2024-09-21T15:16:54.533 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:54.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:54.608 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:54.608+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:54.653 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:54.654 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:54.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:54.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:54.708 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:54.709+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:54.809 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:54.808+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:622 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:54.813 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.437733805000001, 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033831148999999998, 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 622 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:54.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374830+0000", 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.374818+0000", 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999886 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375659+0000", 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084095399999999999 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375676+0000", 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6812000000000001e-05 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375684+0000", 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0900000000000005e-06 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375731+0000", 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6737000000000003e-05 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375824+0000", 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3350000000000006e-05 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.375959+0000", 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013498500000000001 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.407620+0000", 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031660559999999997 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408527+0000", 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00090730699999999999 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408600+0000", 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3011999999999998e-05 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.408661+0000", 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0785000000000002e-05 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.354753146, 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.819 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:16:54.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:16:54.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.824 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.306868889, 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.825 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:16:54.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:16:54.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.830 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.258697043, 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:16:54.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:16:54.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:16:54.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:16:54.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.900527941, 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:16:54.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:16:54.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.842 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.7345440700000001, 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:16:54.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.549038801, 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:54.848 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:16:54.853 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.57039996999999998, 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:16:54.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:16:54.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.56961509300000002, 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.859 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:16:54.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:16:54.864 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:25 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243249+0000", 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.56931416000000001, 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036462078000000002, 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 25 2024-09-21T15:16:54.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243249+0000", 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243249+0000", 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243248+0000", 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243449+0000", 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000200476 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243470+0000", 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1362999999999999e-05 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243480+0000", 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2860000000000005e-06 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.265699+0000", 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022219398000000001 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.265827+0000", 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127971 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.266078+0000", 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025141999999999997 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279609+0000", 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013531052999999999 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279647+0000", 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7542000000000001e-05 2024-09-21T15:16:54.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279676+0000", 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9238999999999999e-05 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279712+0000", 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5264e-05 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.56907496899999999, 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:16:54.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:16:54.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.56887255299999995, 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.876 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:16:54.881 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.568676348, 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:16:54.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:16:54.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.56848222500000001, 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:16:54.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:54.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:16:54.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:16:54.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51403500400000002, 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.892 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:16:54.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.898 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51390255699999998, 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.899 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51375117800000003, 2024-09-21T15:16:54.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:16:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:16:54.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:16:54.909 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51362380699999999, 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:16:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:16:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.37882380700000001, 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:16:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.24179772299999999, 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:16:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:16:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:16:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:54.925 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:54.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:54.927 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:54.927 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:54.927 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:54.928 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:54.925+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:54.936 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:54.963 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:54.994 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:55.026 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:55.025+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:55.050 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.050 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:55.127 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:55.128+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:55.170 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.170 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:55.229 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:55.228+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:55.301 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.301 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:55.354 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:55.356+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:55.430 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:55.432+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:55.463 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.464 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:55.532 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:55.529+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:55.590 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.590 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:55.633 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:55.633+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:55.710 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.710 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.720 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:55.733 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:55.732+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:55.834 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:55.836+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:55.835 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.835 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.846 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:55.935 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:55.933+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:55.950 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:55.950 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:55.960 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:56.036 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:56.036+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:56.069 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.069 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:56.137 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:56.136+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:56.196 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.196 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:56.237 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:56.236+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:56.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:16:56.304 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.304 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.314 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:56.337 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:56.336+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:56.408 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.408 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:56.438 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:56.440+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:56.527 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.527 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:56.539 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:56.537+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:56.639 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:56.640+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:56.644 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.644 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.654 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:56.741 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:56.741+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:56.763 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.764 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:56.842 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:56.841+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:56.884 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.884 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:56.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:56.942 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:56.941+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:56.997 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:56.997 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:57.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:16:57.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:57.043 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:57.041+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:57.114 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:57.114 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:57.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:57.144 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:57.145+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:57.156 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:57.179 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:57.200 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:57.222 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:57.224 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:57.224 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:57.224 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:57.224 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:57.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:57.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:57.245 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:57.245+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:57.271 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:57.278 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:16:57.340 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:57.340 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:57.340 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:57.340 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:57.340 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:57.346 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:57.345+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:57.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:16:57.447 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:57.448+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:57.473 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:57.473 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:57.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 99.690213736000004, 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:16:57.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:16:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:16:57.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:624 2.7 e57/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 57'33, mlcod=57'31)", 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.051529650000001, 2024-09-21T15:16:57.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029592388000000001, 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420904+0000", 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421288+0000", 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000384674 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421318+0000", 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0017000000000001e-05 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421332+0000", 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421441+0000", 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109409 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421481+0000", 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0125e-05 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450452+0000", 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028971001 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450498+0000", 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5037999999999998e-05 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.484 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.013276513999999, 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:16:57.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.488 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.685257145, 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:57.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:16:57.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:16:57.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:16:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.634095702, 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:16:57.494 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.546049736000001, 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:16:57.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:16:57.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:16:57.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:658 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'43, mlcod=57'41)", 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.325985339000001, 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031302035999999998, 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.505 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146448+0000", 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146818+0000", 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036965199999999999 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146845+0000", 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7525000000000002e-05 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146858+0000", 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2703e-05 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146954+0000", 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6186000000000004e-05 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146992+0000", 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177692+0000", 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030700439999999999 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177751+0000", 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9271000000000002e-05 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:674 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'7, mlcod=50'5)", 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:57.509 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.006920739, 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030213265, 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:16:57.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465512+0000", 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465905+0000", 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392972 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465940+0000", 2024-09-21T15:16:57.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5026e-05 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465954+0000", 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3256e-05 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466019+0000", 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5701e-05 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466046+0000", 2024-09-21T15:16:57.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6658000000000001e-05 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495689+0000", 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029642966999999999 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495727+0000", 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8352000000000001e-05 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.965517956999999, 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.517 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:684 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.784075360999999, 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028865808999999999, 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:57.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688359+0000", 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688358+0000", 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688856+0000", 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000498044 2024-09-21T15:16:57.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688876+0000", 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.931e-05 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688886+0000", 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0814e-05 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688948+0000", 2024-09-21T15:16:57.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1636000000000004e-05 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.688980+0000", 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2020999999999997e-05 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717144+0000", 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028163898999999999 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.717225+0000", 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1322000000000005e-05 2024-09-21T15:16:57.521 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:688 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'61, mlcod=58'59)", 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.704282383000001, 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030671810000000001, 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768151+0000", 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768527+0000", 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037648899999999998 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768553+0000", 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6327999999999999e-05 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768565+0000", 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1467999999999999e-05 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768687+0000", 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121721 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768746+0000", 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200999999999998e-05 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798766+0000", 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030020013000000002 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798824+0000", 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:690 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'51, mlcod=58'49)", 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.663614256000001, 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029415891, 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808821+0000", 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.808819+0000", 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809316+0000", 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049685199999999999 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809333+0000", 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7133999999999999e-05 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809340+0000", 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9979999999999997e-06 2024-09-21T15:16:57.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809443+0000", 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102642 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.809479+0000", 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5539e-05 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838194+0000", 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028715469 2024-09-21T15:16:57.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.838236+0000", 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2283000000000002e-05 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.559273275000001, 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:16:57.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:16:57.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:16:57.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.996489505, 2024-09-21T15:16:57.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:16:57.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:16:57.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:16:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:725 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.950867409000001, 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029546688000000002, 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.540 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 725 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521567+0000", 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.521565+0000", 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081585200000000005 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522413+0000", 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2228999999999999e-05 2024-09-21T15:16:57.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522431+0000", 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8065999999999999e-05 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522491+0000", 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9710000000000003e-05 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522591+0000", 2024-09-21T15:16:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010077300000000001 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.522752+0000", 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000160963 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.545613+0000", 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02286036 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551049+0000", 2024-09-21T15:16:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005436771 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551086+0000", 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6477999999999999e-05 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.551114+0000", 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8136e-05 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.913780731999999, 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:16:57.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:16:57.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.871661458, 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:16:57.551 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:16:57.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:16:57.556 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:57 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'112, mlcod=62'110)", 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.765785304, 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030174240000000001, 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:16:57.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706648+0000", 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707026+0000", 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037793499999999998 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707044+0000", 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7445e-05 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707051+0000", 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7749999999999993e-06 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707107+0000", 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5374000000000002e-05 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707130+0000", 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3283000000000001e-05 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736698+0000", 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029568046000000001 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736824+0000", 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012566899999999999 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.725170957, 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.561 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:16:57.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:16:57.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.6288655350000001, 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.565 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:16:57.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.569 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:57.570 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records in 2024-09-21T15:16:57.570 INFO:tasks.workunit.client.0.smithi146.stderr:64+0 records out 2024-09-21T15:16:57.571 INFO:tasks.workunit.client.0.smithi146.stderr:67108864 bytes (67 MB, 64 MiB) copied, 3.33216 s, 20.1 MB/s 2024-09-21T15:16:57.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false timeout 10 rbd bench testrbdnbd17425 --io-type write --io-size=1024 --io-total=1024 2024-09-21T15:16:57.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ timeout 10 rbd bench testrbdnbd17425 --io-type write --io-size=1024 --io-total=1024 2024-09-21T15:16:57.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:16:57.572 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:57.572+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:57.637 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:16:57.634+0000 7f35b37fe700 -1 librbd::exclusive_lock::ImageDispatch: 0x7f3598000e00 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-09-21T15:16:57.638 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: failed to flush: (30) Read-only file system 2024-09-21T15:16:57.638 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:16:57.634+0000 7f35b37fe700 -1 librbd::io::AioCompletion: 0x5649c7b2f990 fail: (30) Read-only file system 2024-09-21T15:16:57.638 INFO:tasks.workunit.client.0.smithi146.stderr:bench failed: (30) Read-only file system 2024-09-21T15:16:57.650 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:57.650 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 18450 2024-09-21T15:16:57.650 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:16:57.650 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18450 2024-09-21T15:16:57.650 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:57.650 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:57.650 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:57.652 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:57.652 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:57.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:57.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:57.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:16:57.672 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:57.669+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:57.725 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:57.725 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:57.726 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:16:57.726 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:57.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:16:57.763 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:57.763 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18450\b' 2024-09-21T15:16:57.763 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18450\b' 2024-09-21T15:16:57.773 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:57.772+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:57.832 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:57.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:57.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18450\b' 2024-09-21T15:16:57.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18450\b' 2024-09-21T15:16:57.840 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:57.841 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:57.846 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:57.846 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:16:57.846 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:16:57.846 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:16:57.846 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:57.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:16:57.874 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:57.873+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.138404451, 2024-09-21T15:16:57.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:16:57.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:16:57.963 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.57597994900000005, 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:16:57.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:16:57.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:16:57.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.56524344800000004, 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:16:57.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:16:57.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.971 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.52078661400000004, 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:16:57.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:16:57.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:16:57.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.975 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51300981199999995, 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:16:57.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.47383614099999999, 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:16:57.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:16:57.983 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.46710378000000002, 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.43305228800000001, 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.42292630399999998, 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:16:57.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:16:57.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:16:57.996 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:16:57.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:16:57.998 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:57.996+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:58.098 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:58.097+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:58.114 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:58.114 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:58.115 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:58.115 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:58.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:16:58.199 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:58.197+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:58.230 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:58.231 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:58.231 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:58.231 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:58.231 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:58.240 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:16:58.300 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:58.300+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:58.342 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:16:58.342 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd bench testrbdnbd17425 --io-type write --io-size=1024 --io-total=1024 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.772294367000001, 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.399 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:16:58.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.403 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.731847775, 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.407 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.65614643, 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.609526033, 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.412 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.416 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.558909409, 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:16:58.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.470555937, 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.421 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:16:58.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:16:58.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.981425067, 2024-09-21T15:16:58.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:16:58.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:16:58.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.266544225000001, 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:16:58.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.221837399, 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.434 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:16:58.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.178229968, 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:16:58.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.081921391, 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.447 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 12.920811992000001, 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:16:58.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.451 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 12.875481521999999, 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:16:58.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 12.796258139000001, 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.456 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:16:58.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:16:58.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:16:58.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.9627851449999998, 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:16:58.460 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.464 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.9106196899999999, 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:16:58.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:16:58.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.8658287790000001, 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:16:58.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:16:58.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 3.8263057109999998, 2024-09-21T15:16:58.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:16:58.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.093168133, 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:16:58.481 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.0588843610000001, 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:16:58.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:16:58.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.486 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:58.488 INFO:tasks.workunit.client.0.smithi146.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-21T15:16:58.488 INFO:tasks.workunit.client.0.smithi146.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-21T15:16:58.488 INFO:tasks.workunit.client.0.smithi146.stdout:elapsed: 0 ops: 1 ops/sec: 250.002 bytes/sec: 250 KiB/s 2024-09-21T15:16:58.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:16:58.489 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:58.485+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:16:58.490 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:16:58.589 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:58.589+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:58.603 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:58.603 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:58.603 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:58.603 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:58.611 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:16:58.611 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:16:58.611 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:16:58.611 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:16:58.612 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:16:58.612 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:16:58.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:16:58.690 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:58.692+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:58.696 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18563 2024-09-21T15:16:58.696 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18563 2024-09-21T15:16:58.696 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18563 -C rbd-nbd 2024-09-21T15:16:58.706 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:16:58.706 INFO:tasks.workunit.client.0.smithi146.stdout: 18563 ? 00:00:00 rbd-nbd 2024-09-21T15:16:58.707 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device testrbdnbd17425 18563 2024-09-21T15:16:58.707 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=testrbdnbd17425 2024-09-21T15:16:58.707 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18563 2024-09-21T15:16:58.707 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17425 2024-09-21T15:16:58.708 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:16:58.708 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:16:58.708 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:16:58.709 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:16:58.711 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:16:58.711 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:16:58.711 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17425 2024-09-21T15:16:58.722 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:58.722 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:58.722 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:58.722 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:58.722 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:58.732 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:16:58.790 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:58.789+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:58.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:16:58.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18563\b' 2024-09-21T15:16:58.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18563\b' 2024-09-21T15:16:58.841 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:58.841 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:58.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:58.841 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:58.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.841 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:16:58.841 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 95.022529614000007, 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034430189999999999, 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819481+0000", 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819480+0000", 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819497+0000", 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6993000000000001e-05 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819522+0000", 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819551+0000", 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9085999999999999e-05 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819690+0000", 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013858699999999999 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.819721+0000", 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1389000000000001e-05 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822832+0000", 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0031111559999999999 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:23.822924+0000", 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4208.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 94.012973029999998, 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001412253, 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:58.846 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829038+0000", 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829037+0000", 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829045+0000", 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4300000000000006e-06 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829079+0000", 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3636000000000001e-05 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829097+0000", 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8493000000000001e-05 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829175+0000", 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7150999999999994e-05 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.829219+0000", 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4589000000000003e-05 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830388+0000", 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011690450000000001 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:15:24.830450+0000", 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1501999999999995e-05 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.3 44/43 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 18'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@18'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.339751606, 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00045114500000000002, 2024-09-21T15:16:58.850 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502259+0000", 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502258+0000", 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502295+0000", 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7391000000000002e-05 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502322+0000", 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:16:58.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502346+0000", 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3136999999999999e-05 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502416+0000", 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.0100999999999998e-05 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502427+0000", 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.135e-05 2024-09-21T15:16:58.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:21.502710+0000", 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00028310800000000001 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:58.854 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:58.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:16:58.889 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:58.890 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18563\b' 2024-09-21T15:16:58.891 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:16:58.891 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18563\b' 2024-09-21T15:16:58.891 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:58.892+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:58.900 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:16:58.900 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:16:58.992 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:58.989+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:59.005 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:59.006 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:59.006 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:16:59.006 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:59.015 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:16:59.093 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:59.093+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:59.124 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:59.124 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:16:59.124 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:16:59.124 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:16:59.125 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:59.134 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:16:59.194 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:59.193+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:59.244 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:16:59.244 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.139127471999998, 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:16:59.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.139076428999999, 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.249 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:16:59.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:59.253 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.123456886, 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:16:59.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:16:59.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.123430132999999, 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:16:59.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:16:59.261 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.95438203, 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.262 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:16:59.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.939237173, 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:16:59.268 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.273 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.910090357, 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:16:59.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:16:59.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:59.279 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.900763452, 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:16:59.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:16:59.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:16:59.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:16:59.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.8731273, 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.285 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:16:59.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:16:59.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.862049595, 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:59.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:16:59.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.827280048, 2024-09-21T15:16:59.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:16:59.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:16:59.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:16:59.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:16:59.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.938755966, 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.302 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:16:59.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.928500898999999, 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:16:59.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:16:59.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:59.313 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.896453901999999, 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:16:59.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:16:59.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.884241949, 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.319 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:16:59.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:16:59.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:16:59.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.850345990999999, 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:16:59.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:16:59.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:16:59.330 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.811301122, 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.950857995, 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.336 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:16:59.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.342 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.9392804880000001, 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:16:59.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:16:59.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:16:59.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:16:59.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.90486972, 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:16:59.348 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:16:59.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:16:59.353 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:16:59.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:16:59.354 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:16:59.354 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:16:59.354 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T15:16:59.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-09-21T15:16:59.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:16:59.357 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:16:59.356+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:16:59.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:16:59.402 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:16:59.402 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd snap create rbd/testrbdnbd17425@snap 2024-09-21T15:16:59.457 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:16:59.456+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:16:59.524 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:16:59.525 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:59.525 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:59.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:16:59.551 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:16:59.558 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:16:59.557+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:16:59.582 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:16:59.603 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:16:59.625 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:16:59.633 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:59.633 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:59.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:16:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:16:59.650 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:16:59.650 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:16:59.650 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:16:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:16:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:16:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:16:59.661 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:59.660+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:16:59.700 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-09-21T15:16:59.700+0000 7f6b15ef9700 -1 mon.a@0(leader).osd e64 definitely_dead 0 2024-09-21T15:16:59.753 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:59.753 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:59.760 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:16:59.757+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:16:59.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:16:59.861 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:16:59.861+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:16:59.914 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:16:59.914 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:16:59.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:16:59.962 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:16:59.964+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:00.031 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.031 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:00.090 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:00.092+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:00.145 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.145 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:00.164 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:00.164+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:00.257 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.257 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.265 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:00.264+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:00.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:00.309 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-09-21T15:17:00.311 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:17:00.327 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T15:17:00.366 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:00.365+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:00.407 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.408 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:00.467 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:00.465+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:00.522 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.522 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:00.643 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:00.644+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:00.744 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:00.744+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:00.759 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.759 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.769 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:00.845 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:00.844+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:00.870 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.870 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:00.946 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:00.948+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:00.986 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:00.986 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:00.996 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:01.047 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:01.048+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:01.106 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:01.106 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:01.115 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:01.160 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:01.161+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:01.224 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:01.224 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:01.234 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:01.248 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:01.248+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:01.342 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:01.343 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:01.349 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:01.348+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:01.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:01.450 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:01.452+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:01.458 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:01.458 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:01.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:01.551 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:01.552+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:01.569 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:01.569 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:01.580 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:01.652 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:01.653+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:01.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:01.680 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:01.681 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:01.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:01.752 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:01.753+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:01.812 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:01.822 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:01.822 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:01.822 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:01.822 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:01.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:01.834 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:01.854 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:01.853+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:01.857 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:01.878 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:01.900 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:01.924 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:01.925 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:01.925 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:01.926 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:01.926 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:01.926 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:01.926 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:01.934 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:01.934 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:01.934 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:01.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:01.955 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:01.953+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:02.056 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:02.056 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:02.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:02.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:02.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.056 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:02.056 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.596712951000001, 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:02.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:02.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:02.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:02.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.548828694000001, 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.062 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:02.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:02.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.500656847999998, 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:02.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:02.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:02.073 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.142487746, 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:02.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:02.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:02.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.976503874999999, 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:02.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:02.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.790998605999999, 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:02.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:02.090 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.812359775, 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:02.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:02.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:02.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:02.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8115748979999999, 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8110347740000003, 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:02.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:02.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:02.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8108323579999999, 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:02.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8106361529999999, 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.113 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:02.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:02.118 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8104420299999999, 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:02.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.7559948089999997, 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.124 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:02.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:02.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.7558623620000002, 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:02.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:02.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.7557109830000002, 2024-09-21T15:17:02.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:02.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.7555836119999997, 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.6207836120000003, 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:02.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.4837575279999999, 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:02.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:02.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.3084606240000003, 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:02.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.2581680989999997, 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.169 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:02.170 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:02.053+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:02.171 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:02.169+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:02.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:02.272 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:02.272+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:02.285 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:02.286 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:02.286 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:02.286 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:02.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:02.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:17:02.345 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T15:17:02.361 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425@snap 2024-09-21T15:17:02.361 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:02.362 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:02.364 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:02.364 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:02.367 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:02.367 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:02.367 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425@snap 2024-09-21T15:17:02.372 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:02.373+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:02.409 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:02.409 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:02.409 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:02.409 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:02.409 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:02.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:02.473 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:02.473+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:02.526 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:02.526 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:02.526 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:02.526 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:02.527 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:02.527 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 104.758328249, 2024-09-21T15:17:02.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:02.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.546 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:624 2.7 e57/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 57'33, mlcod=57'31)", 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.119644163, 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029592388000000001, 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420904+0000", 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421288+0000", 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000384674 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421318+0000", 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0017000000000001e-05 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421332+0000", 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-21T15:17:02.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421441+0000", 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109409 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421481+0000", 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0125e-05 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450452+0000", 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028971001 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450498+0000", 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5037999999999998e-05 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:02.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.081391026999999, 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:02.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:02.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.753371657999999, 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:02.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:02.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:02.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:02.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.702210215000001, 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.562 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:02.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:02.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:02.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:02.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:02.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:02.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:02.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:02.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.614164249000002, 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:02.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:02.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:658 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.394099852, 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031302035999999998, 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146448+0000", 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146818+0000", 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036965199999999999 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146845+0000", 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7525000000000002e-05 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146858+0000", 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2703e-05 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146954+0000", 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6186000000000004e-05 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146992+0000", 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177692+0000", 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030700439999999999 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177751+0000", 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9271000000000002e-05 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.584 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:674 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'7, mlcod=50'5)", 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.075035251999999, 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030213265, 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465512+0000", 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465905+0000", 2024-09-21T15:17:02.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392972 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465940+0000", 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5026e-05 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465954+0000", 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3256e-05 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466019+0000", 2024-09-21T15:17:02.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5701e-05 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466046+0000", 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6658000000000001e-05 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495689+0000", 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029642966999999999 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495727+0000", 2024-09-21T15:17:02.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8352000000000001e-05 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.033632470000001, 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:688 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'61, mlcod=58'59)", 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.772396896, 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030671810000000001, 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768151+0000", 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768527+0000", 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037648899999999998 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768553+0000", 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6327999999999999e-05 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768565+0000", 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1467999999999999e-05 2024-09-21T15:17:02.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768687+0000", 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121721 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768746+0000", 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200999999999998e-05 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798766+0000", 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030020013000000002 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798824+0000", 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.627387788, 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.597 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:02.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:02.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:02.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.064604018000001, 2024-09-21T15:17:02.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.606 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.981895245, 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:02.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:02.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:02.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:02.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:02.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.939775971, 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:02.612 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:57 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'112, mlcod=62'110)", 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.8338998169999998, 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030174240000000001, 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706648+0000", 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707026+0000", 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037793499999999998 2024-09-21T15:17:02.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707044+0000", 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7445e-05 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707051+0000", 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7749999999999993e-06 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707107+0000", 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5374000000000002e-05 2024-09-21T15:17:02.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707130+0000", 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3283000000000001e-05 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736698+0000", 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029568046000000001 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736824+0000", 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012566899999999999 2024-09-21T15:17:02.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.7932854699999998, 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:02.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:02.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:02.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:02.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.6969800480000004, 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1468095759999999, 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.630 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:02.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:02.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:02.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.0939710519999997, 2024-09-21T15:17:02.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:02.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.0481295399999997, 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:02.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:02.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:02.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:02.649 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:02.651 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18665 2024-09-21T15:17:02.651 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18665 2024-09-21T15:17:02.651 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18665 -C rbd-nbd 2024-09-21T15:17:02.651 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:02.651 INFO:tasks.workunit.client.0.smithi146.stdout: 18665 ? 00:00:00 rbd-nbd 2024-09-21T15:17:02.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:02.652 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:02.652+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:02.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device testrbdnbd17425@snap 18665 2024-09-21T15:17:02.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=testrbdnbd17425@snap 2024-09-21T15:17:02.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18665 2024-09-21T15:17:02.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17425@snap 2024-09-21T15:17:02.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:02.654 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:02.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:02.654 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:02.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:02.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:02.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17425@snap 2024-09-21T15:17:02.734 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18665\b' 2024-09-21T15:17:02.734 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:02.734 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18665\b' 2024-09-21T15:17:02.752 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:02.752+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:02.803 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:02.804 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:02.804 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18665\b' 2024-09-21T15:17:02.804 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18665\b' 2024-09-21T15:17:02.806 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:02.806 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:02.806 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:02.806 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:02.809 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:02.810 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:02.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:02.853 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:02.852+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:02.919 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:02.919 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:02.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:02.953 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:02.953+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:03.054 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:03.053+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:03.084 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:03.084 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.26333232899999, 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.700907827, 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:03.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:03.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:03.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6901713259999998, 2024-09-21T15:17:03.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6457144919999998, 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6379376900000002, 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:03.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5987640189999999, 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:03.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5920316579999998, 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:03.110 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5579801660000001, 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.547854182, 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:03.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:03.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:03.122 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:03.123 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:03.155 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:03.156+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:03.233 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:03.233 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:03.233 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:03.233 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:03.243 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:03.255 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:03.253+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:03.311 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:03.312 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd snap ls rbd/testrbdnbd17425 2024-09-21T15:17:03.312 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-21T15:17:03.339 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:17:03.350 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:03.351 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:03.351 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:03.356 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:03.356+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:03.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:03.457 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:03.456+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:03.463 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:03.463 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:03.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.834339432, 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:03.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:03.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.793892840000002, 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:03.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:03.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.718191494999999, 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:03.472 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:03.476 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.671571098000001, 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:03.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.620954474000001, 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.484 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.532601002, 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.043470131999999, 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:03.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.32858929, 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.493 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:03.497 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.283882464000001, 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:03.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:03.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.501 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.240275033, 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:03.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.143966456000001, 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:03.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:03.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.982857057, 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.937526587000001, 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:03.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.858303203999998, 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.518 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:03.522 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.0248302099999993, 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9726647550000003, 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:03.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9278738440000005, 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.8883507759999993, 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.538 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 6.1552131980000002, 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:03.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.542 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 6.120929426, 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:03.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:03.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:03.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:03.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:03.547 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.547 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.547 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.547 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.547 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.547 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:03.548 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:03.557 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:03.556+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:03.655 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:03.655 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:03.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5442.0:1 2.5 2.eaaa11d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e68)", 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.364563+0000", 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.29202181500000002, 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.29206136500000002, 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "waiting for peered", 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5442", 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/600884957", 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 1 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364563+0000", 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364563+0000", 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364561+0000", 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364570+0000", 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.0729999999999994e-06 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364588+0000", 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7193e-05 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364602+0000", 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4018e-05 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364754+0000", 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015271600000000001 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for peered", 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.364772+0000", 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7719000000000001e-05 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 1 2024-09-21T15:17:03.659 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:03.660 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:03.661+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:03.665 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:03.762 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:03.760+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:03.767 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:03.767 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:03.767 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:03.767 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:03.767 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:03.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:03.862 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:03.861+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:03.883 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:03.883 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:03.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:03.883 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:03.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.883 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:ac59e578:ns::rbd_data.137aae78387e.0000000000000007:head, version: 50'15, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ac59e578:ns::rbd_data.137aae78387e.0000000000000007:head@50'15, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:03.883 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.5417913640000001, 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03268339, 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:03.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339330+0000", 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340709+0000", 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013791540000000001 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340735+0000", 2024-09-21T15:17:03.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5687e-05 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340755+0000", 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0428000000000002e-05 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340873+0000", 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011763 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340890+0000", 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6713000000000001e-05 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.372014+0000", 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031124496000000001 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:b3267759:ns::rbd_data.137aae78387e.000000000000000a:head, version: 50'20, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b3267759:ns::rbd_data.137aae78387e.000000000000000a:head@50'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:03.887 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.4129661179999999, 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033844346999999997, 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468154+0000", 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469522+0000", 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013677920000000001 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469560+0000", 2024-09-21T15:17:03.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8232999999999998e-05 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469584+0000", 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3595999999999999e-05 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469777+0000", 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000192871 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469814+0000", 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7174e-05 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.502000+0000", 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032186335000000003 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head, version: 63'92, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head@63'92, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:03.891 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.2860285579999999, 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032502650000000001, 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596191+0000", 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010984600000000001 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596207+0000", 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5761999999999999e-05 2024-09-21T15:17:03.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596216+0000", 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2410000000000001e-06 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596392+0000", 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017578099999999999 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596413+0000", 2024-09-21T15:17:03.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1267000000000002e-05 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.627596+0000", 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031182781999999999 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head, version: 63'92, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head@63'92, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.191886317, 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035308071000000003, 2024-09-21T15:17:03.895 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689234+0000", 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690873+0000", 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016385620000000001 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690906+0000", 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2920000000000003e-05 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690930+0000", 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4391000000000001e-05 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.691115+0000", 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000184554 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.691153+0000", 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8371999999999998e-05 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.724544+0000", 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033390729000000001 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head, version: 63'100, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head@63'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.728405+0000", 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.152717011, 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030669017999999999, 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.899 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.728405+0000", 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.728405+0000", 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.728403+0000", 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730266+0000", 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001862963 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730303+0000", 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6350000000000003e-05 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730347+0000", 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3838999999999998e-05 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730475+0000", 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000127985 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730484+0000", 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5640000000000002e-06 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.759074+0000", 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028589823 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head, version: 63'100, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head@63'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.793915+0000", 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.0872065209999999, 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030685982000000001, 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.793915+0000", 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.793915+0000", 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.793909+0000", 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999938 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795641+0000", 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017322049999999999 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795680+0000", 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8173000000000002e-05 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795707+0000", 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7345999999999999e-05 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795805+0000", 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7922000000000002e-05 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795819+0000", 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4301000000000001e-05 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.824601+0000", 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028782275 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.906 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head, version: 64'101, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head@64'101, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.863762+0000", 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.0173597240000001, 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031866761, 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.863762+0000", 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.863762+0000", 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.863761+0000", 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865108+0000", 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013471659999999999 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865145+0000", 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7546000000000003e-05 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865184+0000", 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8547000000000002e-05 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865312+0000", 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000128102 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865332+0000", 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0483e-05 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.895629+0000", 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030296613999999999 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.910 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head, version: 64'101, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head@64'101, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.933087+0000", 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.94803501400000001, 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031871006, 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.933087+0000", 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.933087+0000", 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.933086+0000", 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934310+0000", 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0012238139999999999 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934327+0000", 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6557000000000001e-05 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934345+0000", 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8729999999999999e-05 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934529+0000", 2024-09-21T15:17:03.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018341399999999999 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934551+0000", 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2076e-05 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.964958+0000", 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030407136000000001 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head, version: 62'100, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head@62'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.87328656100000002, 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031759030000000001, 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009023+0000", 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011885050000000001 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009049+0000", 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5992000000000001e-05 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009065+0000", 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6008999999999999e-05 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009224+0000", 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000158362 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009241+0000", 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7467999999999999e-05 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.039594+0000", 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030353508000000001 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head, version: 62'100, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head@62'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.918 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.046393+0000", 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.83472853499999999, 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029710481, 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.046393+0000", 2024-09-21T15:17:03.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.046393+0000", 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.046392+0000", 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.047885+0000", 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001493416 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.047923+0000", 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7956e-05 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.047952+0000", 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9326999999999999e-05 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.048054+0000", 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010131799999999999 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.048062+0000", 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0020000000000006e-06 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.076104+0000", 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028042249000000002 2024-09-21T15:17:03.922 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head, version: 62'100, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head@62'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.76854966800000002, 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034266194, 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113722+0000", 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011501860000000001 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113737+0000", 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5199e-05 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113747+0000", 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9650000000000003e-06 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113880+0000", 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013289500000000001 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113897+0000", 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6762e-05 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.146839+0000", 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032941830999999998 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.926 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.69665163600000002, 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032588833999999997, 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:03.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184469+0000", 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186107+0000", 2024-09-21T15:17:03.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016386580000000001 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186149+0000", 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1878999999999997e-05 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186192+0000", 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3130000000000002e-05 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186334+0000", 2024-09-21T15:17:03.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142235 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186359+0000", 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4260000000000002e-05 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.217059+0000", 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030700531999999999 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.660280113, 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036223334000000003, 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:03.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220840+0000", 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222163+0000", 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001322939 2024-09-21T15:17:03.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222190+0000", 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7030999999999999e-05 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222219+0000", 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8646e-05 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222352+0000", 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013276799999999999 2024-09-21T15:17:03.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222375+0000", 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3467000000000001e-05 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.257065+0000", 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034689910999999997 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.61825026400000005, 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:03.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.54402504799999996, 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:03.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.533625609, 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:03.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:03.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.407304+0000", 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.473817773, 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030614235, 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.407304+0000", 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.407304+0000", 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.407303+0000", 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.408940+0000", 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016372839999999999 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.408975+0000", 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4714e-05 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409002+0000", 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7764999999999999e-05 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409101+0000", 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9152000000000002e-05 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409119+0000", 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7104000000000001e-05 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.437918+0000", 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028799874 2024-09-21T15:17:03.950 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.471318603, 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:03.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.954 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.470145+0000", 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.41097668300000001, 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032019508000000002, 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.470145+0000", 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.470145+0000", 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.470144+0000", 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471431+0000", 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00128681 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471454+0000", 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3244e-05 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471468+0000", 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4099e-05 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471543+0000", 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4993000000000005e-05 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471560+0000", 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.715e-05 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502165+0000", 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030604612 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.958 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.409108535, 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:03.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:03.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:03.962 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:03.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:03.964 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:03.965 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:03.965+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:04.065 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:04.065+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:04.118 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:04.119 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:04.119 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:04.119 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:04.119 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:04.129 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:04.137 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:04.156 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:04.165 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:04.164+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:04.177 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:04.198 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:04.222 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:04.224 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:04.240 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:04.240 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:04.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:04.266 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:04.268+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:04.352 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:04.352 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:04.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:04.406 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:04.408+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:04.467 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:04.465+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:04.482 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:04.482 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:04.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:04.569 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:04.568+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:04.598 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:04.598 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:04.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:04.669 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:04.668+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:04.717 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:04.717 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:04.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:04.770 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:04.769+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:04.871 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:04.869+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:04.886 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:04.886 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:04.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:04.972 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:04.972+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:05.002 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.002 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:05.073 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:05.073+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:05.166 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.166 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.174 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:05.173+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:05.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:05.230 INFO:tasks.workunit.client.0.smithi146.stderr:+ SNAPID=4 2024-09-21T15:17:05.231 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd17425 2024-09-21T15:17:05.231 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:05.231 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:05.234 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:05.234 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:05.236 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:05.236 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:05.236 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd17425 2024-09-21T15:17:05.275 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:05.273+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:05.285 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.286 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:05.327 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:05.327 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:05.327 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T15:17:05.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:17:05.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:17:05.375 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:05.373+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:05.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:05.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:05.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:05.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:05.403 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:05.404 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:05.422 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.422 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:05.475 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:05.476+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:05.486 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18767 2024-09-21T15:17:05.486 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18767 2024-09-21T15:17:05.486 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18767 -C rbd-nbd 2024-09-21T15:17:05.498 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:05.498 INFO:tasks.workunit.client.0.smithi146.stdout: 18767 ? 00:00:00 rbd-nbd 2024-09-21T15:17:05.499 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device '--snap-id 4 testrbdnbd17425' 18767 2024-09-21T15:17:05.499 INFO:tasks.workunit.client.0.smithi146.stderr:+ local 'args=--snap-id 4 testrbdnbd17425' 2024-09-21T15:17:05.499 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18767 2024-09-21T15:17:05.499 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 testrbdnbd17425 2024-09-21T15:17:05.499 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:05.499 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:05.501 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:05.501 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:05.502 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:05.503 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:05.503 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 testrbdnbd17425 2024-09-21T15:17:05.550 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.550 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.561 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:05.576 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:05.576+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:05.617 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:05.617 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18767\b' 2024-09-21T15:17:05.617 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18767\b' 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 70, 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:17:05.199630+0000", 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:17:00.287583+0000", 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:17:05.659 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5804032, 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:17:05.660 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:05.661 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:05.662 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:17:05.663 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:05.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:17:05.666 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:05.667 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "65", 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 4, 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 65, 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:05.669 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:05.670 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:05.671 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.25, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.25, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2.25, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2.25, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.672 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "new_default_pool17425", 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:16:34.640599+0000", 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:05.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "54", 2024-09-21T15:17:05.674 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 54, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:05.675 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:05.676 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 3.0099999904632568, 2024-09-21T15:17:05.677 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 3.0099999904632568, 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2.5, 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2.5, 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:17:05.678 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 68, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:05.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.427032470703125, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:05.684 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.686 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:05.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:17:05.688 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:05.689 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:17:05.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:05.694 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:17:05.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:05.698 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:17:05.699 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 69, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:17:05.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:05.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:17:05.704 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:05.710 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:05.711 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:17:05.712 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.1", 2024-09-21T15:17:05.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: 4, 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: 5 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [], 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:17:05.714 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:17:05.715 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:17:05.716 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:17:05.716 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:17:05.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:05.716 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:05.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:05.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18767\b' 2024-09-21T15:17:05.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18767\b' 2024-09-21T15:17:05.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:05.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:05.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:05.718 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.719 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.719 INFO:tasks.thrashosds.thrasher:j is {'epoch': 70, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:17:05.199630+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:17:00.287583+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 3, 'pool_name': 'new_default_pool17425', 'create_time': '2024-09-21T15:16:34.640599+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 68, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.427032470703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.1', 'osds': [4, 5]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:17:05.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:17:05.720 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:05.716+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:05.720 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:05.818 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:05.817+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:05.830 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.830 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.840 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:05.918 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:05.917+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:05.946 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:05.947 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:05.956 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:06.019 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:06.017+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:06.064 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:06.064 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:06.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:06.071 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:17:06.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:06.084 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-21T15:17:05.944220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134808280,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":2646,"num_read_kb":133486,"num_write":704,"num_write_kb":264742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":75497599,"num_keys_recovered":20,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":632,"ondisk_log_size":632,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":514912,"kb_used_data":328200,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":565716128,"statfs":{"total":579820584960,"available":579293315072,"internally_reserved":0,"allocated":336076800,"data_stored":334230328,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4194342,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":610,"num_write":12,"num_write_kb":4099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":50331702,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001378"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":210,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T15:17:00.303121+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:17:00.303121+0000","last_peered":"2024-09-21T15:17:00.303121+0000","last_clean":"2024-09-21T15:17:00.303121+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:17:00.303121+0000","last_undegraded":"2024-09-21T15:17:00.303121+0000","last_fullsized":"2024-09-21T15:17:00.303121+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"64'39","reported_seq":472,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T15:17:00.303288+0000","last_change":"2024-09-21T15:16:55.672969+0000","last_active":"2024-09-21T15:17:00.303288+0000","last_peered":"2024-09-21T15:17:00.303288+0000","last_clean":"2024-09-21T15:17:00.303288+0000","last_became_active":"2024-09-21T15:16:55.272135+0000","last_became_peered":"2024-09-21T15:16:55.272135+0000","last_unstale":"2024-09-21T15:17:00.303288+0000","last_undegraded":"2024-09-21T15:17:00.303288+0000","last_fullsized":"2024-09-21T15:17:00.303288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:58.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":327,"num_read_kb":20640,"num_write":38,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"54'2","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T15:17:03.679960+0000","last_change":"2024-09-21T15:16:55.572914+0000","last_active":"2024-09-21T15:17:03.679960+0000","last_peered":"2024-09-21T15:17:03.679960+0000","last_clean":"2024-09-21T15:17:03.679960+0000","last_became_active":"2024-09-21T15:16:55.273190+0000","last_became_peered":"2024-09-21T15:16:55.273190+0000","last_unstale":"2024-09-21T15:17:03.679960+0000","last_undegraded":"2024-09-21T15:17:03.679960+0000","last_fullsized":"2024-09-21T15:17:03.679960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:38.164122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181755,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"50'4","reported_seq":26,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T15:17:03.679896+0000","last_change":"2024-09-21T15:16:55.569580+0000","last_active":"2024-09-21T15:17:03.679896+0000","last_peered":"2024-09-21T15:17:03.679896+0000","last_clean":"2024-09-21T15:17:03.679896+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:17:03.679896+0000","last_undegraded":"2024-09-21T15:17:03.679896+0000","last_fullsized":"2024-09-21T15:17:03.679896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388624,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T15:17:04.204260+0000","last_change":"2024-09-21T15:17:01.318865+0000","last_active":"2024-09-21T15:17:04.204260+0000","last_peered":"2024-09-21T15:17:04.204260+0000","last_clean":"2024-09-21T15:17:04.204260+0000","last_became_active":"2024-09-21T15:17:01.318630+0000","last_became_peered":"2024-09-21T15:17:01.318630+0000","last_unstale":"2024-09-21T15:17:04.204260+0000","last_undegraded":"2024-09-21T15:17:04.204260+0000","last_fullsized":"2024-09-21T15:17:04.204260+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:40:26.824791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"64'20","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T15:17:04.204215+0000","last_change":"2024-09-21T15:17:01.739071+0000","last_active":"2024-09-21T15:17:04.204215+0000","last_peered":"2024-09-21T15:17:04.204215+0000","last_clean":"2024-09-21T15:17:04.204215+0000","last_became_active":"2024-09-21T15:17:01.316522+0000","last_became_peered":"2024-09-21T15:17:01.316522+0000","last_unstale":"2024-09-21T15:17:04.204215+0000","last_undegraded":"2024-09-21T15:17:04.204215+0000","last_fullsized":"2024-09-21T15:17:04.204215+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:15.523790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":16384,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":36,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T15:17:00.303692+0000","last_change":"2024-09-21T15:16:55.272512+0000","last_active":"2024-09-21T15:17:00.303692+0000","last_peered":"2024-09-21T15:17:00.303692+0000","last_clean":"2024-09-21T15:17:00.303692+0000","last_became_active":"2024-09-21T15:16:55.271874+0000","last_became_peered":"2024-09-21T15:16:55.271874+0000","last_unstale":"2024-09-21T15:17:00.303692+0000","last_undegraded":"2024-09-21T15:17:00.303692+0000","last_fullsized":"2024-09-21T15:17:00.303692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:54:48.789474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046520300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"63'22","reported_seq":185,"reported_epoch":66,"state":"active+recovering","last_fresh":"2024-09-21T15:17:01.645586+0000","last_change":"2024-09-21T15:17:01.330379+0000","last_active":"2024-09-21T15:17:01.645586+0000","last_peered":"2024-09-21T15:17:01.645586+0000","last_clean":"2024-09-21T15:16:59.464401+0000","last_became_active":"2024-09-21T15:17:01.327838+0000","last_became_peered":"2024-09-21T15:17:01.327838+0000","last_unstale":"2024-09-21T15:17:01.645586+0000","last_undegraded":"2024-09-21T15:17:01.645586+0000","last_fullsized":"2024-09-21T15:17:01.645586+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:08.047427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":73,"num_read_kb":8232,"num_write":22,"num_write_kb":20482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,4","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'1","reported_seq":41,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T15:17:00.305426+0000","last_change":"2024-09-21T15:16:55.676403+0000","last_active":"2024-09-21T15:17:00.305426+0000","last_peered":"2024-09-21T15:17:00.305426+0000","last_clean":"2024-09-21T15:17:00.305426+0000","last_became_active":"2024-09-21T15:16:55.274451+0000","last_became_peered":"2024-09-21T15:16:55.274451+0000","last_unstale":"2024-09-21T15:17:00.305426+0000","last_undegraded":"2024-09-21T15:17:00.305426+0000","last_fullsized":"2024-09-21T15:17:00.305426+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:20.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046455,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T15:17:03.680052+0000","last_change":"2024-09-21T15:16:55.271559+0000","last_active":"2024-09-21T15:17:03.680052+0000","last_peered":"2024-09-21T15:17:03.680052+0000","last_clean":"2024-09-21T15:17:03.680052+0000","last_became_active":"2024-09-21T15:16:55.271108+0000","last_became_peered":"2024-09-21T15:16:55.271108+0000","last_unstale":"2024-09-21T15:17:03.680052+0000","last_undegraded":"2024-09-21T15:17:03.680052+0000","last_fullsized":"2024-09-21T15:17:03.680052+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:30.189660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"63'25","reported_seq":134,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T15:17:00.303908+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:17:00.303908+0000","last_peered":"2024-09-21T15:17:00.303908+0000","last_clean":"2024-09-21T15:17:00.303908+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:17:00.303908+0000","last_undegraded":"2024-09-21T15:17:00.303908+0000","last_fullsized":"2024-09-21T15:17:00.303908+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":25165824,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":8203,"num_write":25,"num_write_kb":32770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T15:17:04.204340+0000","last_change":"2024-09-21T15:17:01.318184+0000","last_active":"2024-09-21T15:17:04.204340+0000","last_peered":"2024-09-21T15:17:04.204340+0000","last_clean":"2024-09-21T15:17:04.204340+0000","last_became_active":"2024-09-21T15:17:01.317913+0000","last_became_peered":"2024-09-21T15:17:01.317913+0000","last_unstale":"2024-09-21T15:17:04.204340+0000","last_undegraded":"2024-09-21T15:17:04.204340+0000","last_fullsized":"2024-09-21T15:17:04.204340+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:47:53.254684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043867299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"64'101","reported_seq":677,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T15:17:04.210247+0000","last_change":"2024-09-21T15:17:04.208777+0000","last_active":"2024-09-21T15:17:04.210247+0000","last_peered":"2024-09-21T15:17:04.210247+0000","last_clean":"2024-09-21T15:17:04.210247+0000","last_became_active":"2024-09-21T15:17:04.207478+0000","last_became_peered":"2024-09-21T15:17:04.207478+0000","last_unstale":"2024-09-21T15:17:04.210247+0000","last_undegraded":"2024-09-21T15:17:04.210247+0000","last_fullsized":"2024-09-21T15:17:04.210247+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:06:45.553105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":20971574,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":661,"num_read_kb":29663,"num_write":117,"num_write_kb":45091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T15:17:04.205722+0000","last_change":"2024-09-21T15:17:01.321005+0000","last_active":"2024-09-21T15:17:04.205722+0000","last_peered":"2024-09-21T15:17:04.205722+0000","last_clean":"2024-09-21T15:17:04.205722+0000","last_became_active":"2024-09-21T15:17:01.320694+0000","last_became_peered":"2024-09-21T15:17:01.320694+0000","last_unstale":"2024-09-21T15:17:04.205722+0000","last_undegraded":"2024-09-21T15:17:04.205722+0000","last_fullsized":"2024-09-21T15:17:04.205722+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:22.824314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092366200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'146","reported_seq":649,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-09-21T15:17:03.566278+0000","last_change":"2024-09-21T15:17:04.936285+0000","last_active":"2024-09-21T15:17:04.936285+0000","last_peered":"2024-09-21T15:17:03.566278+0000","last_clean":"2024-09-21T15:16:59.475377+0000","last_became_active":"2024-09-21T15:17:02.335478+0000","last_became_peered":"2024-09-21T15:17:02.335478+0000","last_unstale":"2024-09-21T15:17:03.566278+0000","last_undegraded":"2024-09-21T15:17:03.566278+0000","last_fullsized":"2024-09-21T15:17:02.325704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:02:52.206876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":937,"num_read_kb":33526,"num_write":99,"num_write_kb":49161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":20,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T15:17:04.204400+0000","last_change":"2024-09-21T15:17:01.320994+0000","last_active":"2024-09-21T15:17:04.204400+0000","last_peered":"2024-09-21T15:17:04.204400+0000","last_clean":"2024-09-21T15:17:04.204400+0000","last_became_active":"2024-09-21T15:17:01.320707+0000","last_became_peered":"2024-09-21T15:17:01.320707+0000","last_unstale":"2024-09-21T15:17:04.204400+0000","last_undegraded":"2024-09-21T15:17:04.204400+0000","last_fullsized":"2024-09-21T15:17:04.204400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:29:15.250976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"64'155","reported_seq":371,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T15:17:00.306390+0000","last_change":"2024-09-21T15:16:43.080033+0000","last_active":"2024-09-21T15:17:00.306390+0000","last_peered":"2024-09-21T15:17:00.306390+0000","last_clean":"2024-09-21T15:17:00.306390+0000","last_became_active":"2024-09-21T15:16:43.079469+0000","last_became_peered":"2024-09-21T15:16:43.079469+0000","last_unstale":"2024-09-21T15:17:00.306390+0000","last_undegraded":"2024-09-21T15:17:00.306390+0000","last_fullsized":"2024-09-21T15:17:00.306390+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:51:44.297946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":20971558,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":440,"num_read_kb":16737,"num_write":219,"num_write_kb":53379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134217893,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2530,"num_read_kb":133386,"num_write":543,"num_write_kb":262324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":75497580,"num_keys_recovered":20,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331407360,"data_stored":331350381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":512,"ondisk_log_size":512,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76956,"kb_used_data":45840,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94294884,"statfs":{"total":96636764160,"available":96557961216,"internally_reserved":0,"allocated":46940160,"data_stored":47004190,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.81399999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.77400000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.88100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85108,"kb_used_data":53988,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94286732,"statfs":{"total":96636764160,"available":96549613568,"internally_reserved":0,"allocated":55283712,"data_stored":54807527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.78700000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.67200000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.71199999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.73499999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.53300000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101548,"kb_used_data":70428,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270292,"statfs":{"total":96636764160,"available":96532779008,"internally_reserved":0,"allocated":72118272,"data_stored":72170109,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":1.387},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.81000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.878},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":1.1970000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":1.018},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":1.2350000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.94099999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101492,"kb_used_data":70372,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270348,"statfs":{"total":96636764160,"available":96532836352,"internally_reserved":0,"allocated":72060928,"data_stored":71579741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.81999999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.628}]}]},{"osd":4,"up_from":12,"seq":51539607579,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93332,"kb_used_data":62216,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94278508,"statfs":{"total":96636764160,"available":96541192192,"internally_reserved":0,"allocated":63709184,"data_stored":63206283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.65100000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.748}]},{"osd":2,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.61899999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.78300000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:16:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.88500000000000001}]}]},{"osd":2,"up_from":12,"seq":51539607577,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56476,"kb_used_data":25356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315364,"statfs":{"total":96636764160,"available":96578932736,"internally_reserved":0,"allocated":25964544,"data_stored":25462478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.65900000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.56000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":1.417},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":1.3169999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:16:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":1.536}]},{"osd":5,"last update":"Sat Sep 21 15:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":1.3660000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":1.595}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71319552,"data_stored":71303279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71319552,"data_stored":71303279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25169920,"data_stored":25165840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54525952,"data_stored":54525952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":62930944,"data_stored":62914671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":46141440,"data_stored":46137360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:17:06.086 INFO:tasks.thrashosds.thrasher:Setting 3.1 to [2, 3] 2024-09-21T15:17:06.086 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1', '2', '3'] 2024-09-21T15:17:06.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1 2 3 2024-09-21T15:17:06.120 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:06.120+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:06.179 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:06.179 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:06.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:06.204 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:06.204 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd snap create rbd/ns/testrbdnbd17425@snap 2024-09-21T15:17:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:17:06.221 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:06.221+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:06.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:06.303 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:06.303 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:06.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:06.356 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:06.357+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:06.423 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:06.421+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:06.425 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:06.450 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:06.465 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:06.465 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:06.469 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:06.475 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:06.492 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:06.524 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:06.524+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:06.540 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:06.582 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:06.583 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:06.583 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:06.583 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:06.583 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:06.593 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:06.625 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:06.625+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:06.701 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:06.701 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:06.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:06.701 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:06.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.595385090000001, 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.702 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.595334047000001, 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:06.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:06.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:06.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:06.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:06.711 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.579714504000002, 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:06.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:06.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:06.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:06.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 25.579687751000002, 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.716 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:06.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:06.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.410639648, 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:06.720 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:06.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.726 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.395494791000001, 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:06.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:06.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.731 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:06.732 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.366347975, 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:06.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:06.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:06.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:06.737 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.357021069999998, 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.738 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.739 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:06.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:06.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:06.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.329384917999999, 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:06.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.318307213000001, 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:06.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:06.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:06.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:06.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.283537666000001, 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.758 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:06.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:06.764 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.395013584000001, 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:06.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:06.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.384758517000002, 2024-09-21T15:17:06.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:06.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.775 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.35271152, 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.780 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.340499566999998, 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:06.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:06.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:06.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:06.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:06.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.306603609, 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.267558739999998, 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:06.792 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:06.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:06.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.4071156130000002, 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:06.798 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:06.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.3955381060000001, 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:06.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:06.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:06.810 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.3611273379999993, 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:06.816 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:06.816 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:06.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:06.818 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:06.816+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:06.918 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:06.920+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:06.937 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:06.937 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:06.937 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:06.937 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:06.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:07.019 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:07.017+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:07.051 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:07.051 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:07.051 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:07.051 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:07.051 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:07.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:07.119 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:07.120+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:07.170 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.711092501, 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:07.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:07.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.663208244, 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:07.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.615036398000001, 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.182 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.187 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.256867295999999, 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:07.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.090883425000001, 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:07.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:07.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:07.198 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.905378156000001, 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:07.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:07.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:07.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.926739325, 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.205 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:07.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:07.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:07.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:07.210 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.925954448000001, 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:07.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:07.215 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.925414324, 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.216 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.925211908, 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:07.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:07.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:07.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.226 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.925015703, 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:07.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:07.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:07.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:07.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.92482158, 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.232 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.870374358999999, 2024-09-21T15:17:07.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:07.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:07.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.870241912000001, 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.248 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.870090533000001, 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:07.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:07.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:07.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:07.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:07.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.869963161999999, 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:07.254 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:07.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:07.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:07.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.260 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.735163161999999, 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:07.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:07.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:07.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.598137078000001, 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:07.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:07.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:07.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:07.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.422840173999999, 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:07.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:07.278 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.372547648999999, 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:07.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.284 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:07.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:07.287 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T15:17:07.287 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd17425@snap 2024-09-21T15:17:07.287 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:07.287 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:07.287 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:07.287 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:07.287 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:07.288 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:07.288 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd17425@snap 2024-09-21T15:17:07.288 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.1 pg_upmap mapping to [2,3] 2024-09-21T15:17:07.288 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:07.284+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:07.386 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:07.388+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:07.422 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:07.422 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd ns 2024-09-21T15:17:07.422 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:07.422 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns=ns 2024-09-21T15:17:07.423 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:07.423 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:07.448 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:07.449 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:07.449 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:07.449 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:07.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:07.487 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:07.485+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:07.508 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18866 2024-09-21T15:17:07.509 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18866 2024-09-21T15:17:07.509 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18866 -C rbd-nbd 2024-09-21T15:17:07.516 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:07.516 INFO:tasks.workunit.client.0.smithi146.stdout: 18866 ? 00:00:00 rbd-nbd 2024-09-21T15:17:07.517 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device rbd/ns/testrbdnbd17425@snap 18866 2024-09-21T15:17:07.517 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=rbd/ns/testrbdnbd17425@snap 2024-09-21T15:17:07.517 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18866 2024-09-21T15:17:07.517 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd17425@snap 2024-09-21T15:17:07.517 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:07.517 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:07.519 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:07.519 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:07.521 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:07.521 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:07.521 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd17425@snap 2024-09-21T15:17:07.565 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:07.565 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:07.565 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:07.565 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:07.565 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:07.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:07.588 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:07.588+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:07.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:07.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18866\b' 2024-09-21T15:17:07.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18866\b' 2024-09-21T15:17:07.679 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.896634354, 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:07.680 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:624 2.7 e57/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 57'33, mlcod=57'31)", 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.257950267999998, 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029592388000000001, 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.686 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420904+0000", 2024-09-21T15:17:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421288+0000", 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000384674 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421318+0000", 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0017000000000001e-05 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421332+0000", 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-21T15:17:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421441+0000", 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109409 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421481+0000", 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0125e-05 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450452+0000", 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028971001 2024-09-21T15:17:07.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450498+0000", 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5037999999999998e-05 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.219697132, 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.690 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:07.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:07.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:07.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.891677763000001, 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:07.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.840516319999999, 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:07.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:07.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:07.706 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.752470354, 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:07.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:658 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.532405957000002, 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031302035999999998, 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.712 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146448+0000", 2024-09-21T15:17:07.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146818+0000", 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036965199999999999 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146845+0000", 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7525000000000002e-05 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146858+0000", 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2703e-05 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146954+0000", 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6186000000000004e-05 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146992+0000", 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177692+0000", 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030700439999999999 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177751+0000", 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9271000000000002e-05 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.716 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:674 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'7, mlcod=50'5)", 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.213341357000001, 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030213265, 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465512+0000", 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465905+0000", 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392972 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465940+0000", 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5026e-05 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465954+0000", 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3256e-05 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466019+0000", 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5701e-05 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466046+0000", 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6658000000000001e-05 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495689+0000", 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029642966999999999 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495727+0000", 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8352000000000001e-05 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.171938574999999, 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:688 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'61, mlcod=58'59)", 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.910703001000002, 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030671810000000001, 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768151+0000", 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768527+0000", 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037648899999999998 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768553+0000", 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6327999999999999e-05 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768565+0000", 2024-09-21T15:17:07.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1467999999999999e-05 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768687+0000", 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121721 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768746+0000", 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200999999999998e-05 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798766+0000", 2024-09-21T15:17:07.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030020013000000002 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798824+0000", 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.765693893000002, 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:07.729 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:07.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.733 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.202910122999999, 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.120201349999999, 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.078082076000001, 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:07.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:07.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:57 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'112, mlcod=62'110)", 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.972205922000001, 2024-09-21T15:17:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030174240000000001, 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706648+0000", 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707026+0000", 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037793499999999998 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707044+0000", 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7445e-05 2024-09-21T15:17:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707051+0000", 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7749999999999993e-06 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707107+0000", 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5374000000000002e-05 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707130+0000", 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3283000000000001e-05 2024-09-21T15:17:07.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736698+0000", 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029568046000000001 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736824+0000", 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012566899999999999 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.931591575000001, 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:07.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.835286153, 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:07.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.764 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:07.764 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.285115681000001, 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:07.771 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.232277157, 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:07.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:07.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:07.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:07.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.186435645, 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:07.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:07.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:07.782 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:07.784 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:07.784 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18866\b' 2024-09-21T15:17:07.784 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:07.785 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18866\b' 2024-09-21T15:17:07.785 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:07.785 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:07.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:07.785 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:07.785+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:07.885 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:07.885+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:07.939 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:07.939 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:07.939 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:07.939 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:07.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:07.987 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:07.985+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:08.057 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:08.058 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:08.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:08.114 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:08.116+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:08.176 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:08.176 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:08.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 105.355408965, 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:08.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:08.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:08.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.792984463, 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:08.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:08.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:08.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:08.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.782247962, 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:08.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:08.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:08.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.737791128, 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:08.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:08.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.730014325999999, 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:08.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:08.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.690840655000001, 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:08.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.200 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.684108294, 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:08.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.650056802, 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:08.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:08.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.639930818, 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:08.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.212 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.94316743400000003, 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.80578813100000002, 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:08.217 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:08.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.804964079, 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.225 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:08.226 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:08.227 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:08.227 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd snap ls rbd/ns/testrbdnbd17425 2024-09-21T15:17:08.227 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-21T15:17:08.228 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:08.228 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:08.228+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:08.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:17:08.279 INFO:tasks.workunit.client.0.smithi146.stderr:+ SNAPID=5 2024-09-21T15:17:08.280 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd17425 2024-09-21T15:17:08.280 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:08.280 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:08.282 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:08.283 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:08.285 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:08.285 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:08.285 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd17425 2024-09-21T15:17:08.328 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:08.329+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:08.343 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:08.343 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:08.343 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:08.343 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:08.353 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:08.429 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:08.429+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:08.442 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:08.442 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd ns 2024-09-21T15:17:08.442 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:08.442 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns=ns 2024-09-21T15:17:08.442 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:08.443 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:08.465 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:08.465 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:08.465 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:08.465 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:08.465 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:08.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:08.527 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=18952 2024-09-21T15:17:08.527 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 18952 2024-09-21T15:17:08.527 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 18952 -C rbd-nbd 2024-09-21T15:17:08.530 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:08.529+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:08.537 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:08.537 INFO:tasks.workunit.client.0.smithi146.stdout: 18952 ? 00:00:00 rbd-nbd 2024-09-21T15:17:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device '--snap-id 5 rbd/ns/testrbdnbd17425' 18952 2024-09-21T15:17:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ local 'args=--snap-id 5 rbd/ns/testrbdnbd17425' 2024-09-21T15:17:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=18952 2024-09-21T15:17:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd17425 2024-09-21T15:17:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:08.538 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:08.539 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:08.539 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:08.541 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:08.541 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:08.541 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd17425 2024-09-21T15:17:08.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.952710698000001, 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:08.578 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:08.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.912264105999999, 2024-09-21T15:17:08.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:08.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:08.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.836562761, 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:08.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:08.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.789942364000002, 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:08.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.739325739999998, 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:08.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:08.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.650972268, 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:08.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.161841398, 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.604 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:08.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:08.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:08.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.446960556000001, 2024-09-21T15:17:08.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:08.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:08.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:08.612 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.402253729999998, 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:08.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.358646299, 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:08.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.262337722000002, 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:08.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:08.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.101228323000001, 2024-09-21T15:17:08.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:08.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.055897853000001, 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:08.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:08.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:08.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.976674469999999, 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.143201476, 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.638 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:08.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:08.642 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.091036021000001, 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:08.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:08.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.046245109999999, 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:08.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:08.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.006722042, 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.651 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:08.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:08.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:08.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.273584464000001, 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:08.655 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:08.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.239300692, 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:08.664 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^18952\b' 2024-09-21T15:17:08.665 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^18952\b' 2024-09-21T15:17:08.666 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:08.666 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:08.665+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:08.700 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:08.707 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:08.708 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:08.708 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *18952\b' 2024-09-21T15:17:08.708 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *18952\b' 2024-09-21T15:17:08.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:08.719 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:08.722 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:08.742 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:08.765 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:08.765+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:08.767 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:08.780 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:08.780 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:08.780 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:08.780 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:08.790 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:08.791 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:08.814 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:08.867 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:08.865+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:08.940 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:08.940 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:08.950 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:08.967 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:08.968+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:09.057 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.057 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.067 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:09.068 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:09.068+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:09.169 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:09.168+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:09.173 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.174 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.184 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:09.220 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:09.221 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17425 2024-09-21T15:17:09.221 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:09.221 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:09.223 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:09.224 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:09.226 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:09.226 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:09.227 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17425 2024-09-21T15:17:09.239 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:17:09.270 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:09.269+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:09.289 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.289 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.299 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:09.372 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:09.372+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:09.390 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:09.390 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd ns 2024-09-21T15:17:09.390 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:09.390 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns=ns 2024-09-21T15:17:09.391 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:09.392 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:09.404 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.404 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:09.472 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:09.473+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:09.473 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19016 2024-09-21T15:17:09.473 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19016 2024-09-21T15:17:09.474 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19016 -C rbd-nbd 2024-09-21T15:17:09.485 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:09.485 INFO:tasks.workunit.client.0.smithi146.stdout: 19016 ? 00:00:00 rbd-nbd 2024-09-21T15:17:09.486 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd17425' 19016 2024-09-21T15:17:09.486 INFO:tasks.workunit.client.0.smithi146.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd17425' 2024-09-21T15:17:09.486 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19016 2024-09-21T15:17:09.486 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17425 2024-09-21T15:17:09.487 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:09.487 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:09.487 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:09.488 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:09.490 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:09.490 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:09.490 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17425 2024-09-21T15:17:09.532 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.532 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:09.573 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:09.573+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19016\b' 2024-09-21T15:17:09.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19016\b' 2024-09-21T15:17:09.647 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.647 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:09.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:09.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19016\b' 2024-09-21T15:17:09.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19016\b' 2024-09-21T15:17:09.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:09.663 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:09.664 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:09.702 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:09.704+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:09.766 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.766 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.775 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:09.773+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:09.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:09.875 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:09.876+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:09.887 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:09.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:09.976 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:09.973+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:10.051 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.052 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:10.076 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:10.076+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.166 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17425 --snap snap 2024-09-21T15:17:10.166 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:10.166 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:10.168 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:10.169 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:10.171 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:10.171 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:10.171 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd17425 --snap snap 2024-09-21T15:17:10.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:10.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:10.220+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:10.279 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:10.280+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:10.328 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.328 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:10.347 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:10.347 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd ns 2024-09-21T15:17:10.347 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:10.347 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns=ns 2024-09-21T15:17:10.347 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:10.348 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:10.380 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:10.380+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:10.425 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19080 2024-09-21T15:17:10.425 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19080 2024-09-21T15:17:10.426 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19080 -C rbd-nbd 2024-09-21T15:17:10.431 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:10.431 INFO:tasks.workunit.client.0.smithi146.stdout: 19080 ? 00:00:00 rbd-nbd 2024-09-21T15:17:10.432 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd17425 --snap snap' 19080 2024-09-21T15:17:10.432 INFO:tasks.workunit.client.0.smithi146.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd17425 --snap snap' 2024-09-21T15:17:10.432 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19080 2024-09-21T15:17:10.432 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17425 --snap snap 2024-09-21T15:17:10.432 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:10.432 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:10.433 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:10.433 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:10.435 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:10.435 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:10.435 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd17425 --snap snap 2024-09-21T15:17:10.443 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.443 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:10.498 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:10.500+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:10.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19080\b' 2024-09-21T15:17:10.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:10.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19080\b' 2024-09-21T15:17:10.582 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:10.581+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:10.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:10.606 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:10.606 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19080\b' 2024-09-21T15:17:10.606 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19080\b' 2024-09-21T15:17:10.607 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.607 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.617 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:10.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:10.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:10.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:10.684+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:10.734 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.734 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:10.784 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:10.781+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:10.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:10.848 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.848 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.859 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:10.904 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:10.905+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:10.967 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:10.967 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:10.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:10.984 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:10.984 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:10.984+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:11.004 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:11.022 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:11.043 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:11.066 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:11.081 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:11.081 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:11.081 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:11.081 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:11.086 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:11.085+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:11.090 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:11.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:11.120 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:11.121 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:17:11.121 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:11.122 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:11.124 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:11.124 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:11.127 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:11.127 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:11.127 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:17:11.187 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:11.185+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:11.195 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:11.195 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:11.195 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:11.195 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:11.195 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:11.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:11.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:17:11.286 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:11.287 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:11.287 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:11.287 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:11.288 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:11.288 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:11.288 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:11.288+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:ac59e578:ns::rbd_data.137aae78387e.0000000000000007:head, version: 50'15, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ac59e578:ns::rbd_data.137aae78387e.0000000000000007:head@50'15, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.9793762489999995, 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03268339, 2024-09-21T15:17:11.321 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339330+0000", 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340709+0000", 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013791540000000001 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340735+0000", 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5687e-05 2024-09-21T15:17:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340755+0000", 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0428000000000002e-05 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340873+0000", 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011763 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340890+0000", 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6713000000000001e-05 2024-09-21T15:17:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.372014+0000", 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031124496000000001 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:b3267759:ns::rbd_data.137aae78387e.000000000000000a:head, version: 50'20, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b3267759:ns::rbd_data.137aae78387e.000000000000000a:head@50'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.8505510029999996, 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033844346999999997, 2024-09-21T15:17:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468154+0000", 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469522+0000", 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013677920000000001 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469560+0000", 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8232999999999998e-05 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469584+0000", 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3595999999999999e-05 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469777+0000", 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000192871 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469814+0000", 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7174e-05 2024-09-21T15:17:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.502000+0000", 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032186335000000003 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head, version: 63'92, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head@63'92, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.7236134429999996, 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032502650000000001, 2024-09-21T15:17:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596191+0000", 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010984600000000001 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596207+0000", 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5761999999999999e-05 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596216+0000", 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2410000000000001e-06 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596392+0000", 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017578099999999999 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596413+0000", 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1267000000000002e-05 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.627596+0000", 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031182781999999999 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head, version: 63'92, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head@63'92, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.6294712019999995, 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035308071000000003, 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689234+0000", 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690873+0000", 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016385620000000001 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690906+0000", 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2920000000000003e-05 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690930+0000", 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4391000000000001e-05 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.691115+0000", 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000184554 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.691153+0000", 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8371999999999998e-05 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.724544+0000", 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033390729000000001 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head, version: 63'100, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head@63'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:11.337 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.728405+0000", 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.5903018959999997, 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030669017999999999, 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.728405+0000", 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.728405+0000", 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.728403+0000", 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730266+0000", 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001862963 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730303+0000", 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6350000000000003e-05 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730347+0000", 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3838999999999998e-05 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730475+0000", 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000127985 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.730484+0000", 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5640000000000002e-06 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.759074+0000", 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028589823 2024-09-21T15:17:11.341 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head, version: 63'100, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head@63'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.793915+0000", 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.5247914060000003, 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030685982000000001, 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.793915+0000", 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.793915+0000", 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.793909+0000", 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999938 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795641+0000", 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017322049999999999 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795680+0000", 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8173000000000002e-05 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795707+0000", 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7345999999999999e-05 2024-09-21T15:17:11.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795805+0000", 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7922000000000002e-05 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.795819+0000", 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4301000000000001e-05 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.824601+0000", 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028782275 2024-09-21T15:17:11.345 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head, version: 64'101, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head@64'101, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.863762+0000", 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.454944609, 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031866761, 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.863762+0000", 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.863762+0000", 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.863761+0000", 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:11.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865108+0000", 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013471659999999999 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865145+0000", 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7546000000000003e-05 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865184+0000", 2024-09-21T15:17:11.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8547000000000002e-05 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865312+0000", 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000128102 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.865332+0000", 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0483e-05 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.895629+0000", 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030296613999999999 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head, version: 64'101, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:b723daed:::rbd_data.13743413a48b.0000000000000000:head@64'101, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.933087+0000", 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.3856198989999999, 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031871006, 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.350 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.933087+0000", 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.933087+0000", 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.933086+0000", 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934310+0000", 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0012238139999999999 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934327+0000", 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6557000000000001e-05 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934345+0000", 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8729999999999999e-05 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934529+0000", 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018341399999999999 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.934551+0000", 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2076e-05 2024-09-21T15:17:11.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.964958+0000", 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030407136000000001 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head, version: 62'100, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head@62'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.3108714460000002, 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031759030000000001, 2024-09-21T15:17:11.354 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.007835+0000", 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009023+0000", 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011885050000000001 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009049+0000", 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5992000000000001e-05 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009065+0000", 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6008999999999999e-05 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009224+0000", 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000158362 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.009241+0000", 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7467999999999999e-05 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.039594+0000", 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030353508000000001 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head, version: 62'100, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head@62'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.358 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.046393+0000", 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.2723134199999997, 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029710481, 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.046393+0000", 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.046393+0000", 2024-09-21T15:17:11.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.046392+0000", 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.047885+0000", 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001493416 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.047923+0000", 2024-09-21T15:17:11.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7956e-05 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.047952+0000", 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9326999999999999e-05 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.048054+0000", 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010131799999999999 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.048062+0000", 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0020000000000006e-06 2024-09-21T15:17:11.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.076104+0000", 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028042249000000002 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head, version: 62'100, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head@62'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.2061345530000001, 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034266194, 2024-09-21T15:17:11.362 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113722+0000", 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011501860000000001 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113737+0000", 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5199e-05 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113747+0000", 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9650000000000003e-06 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113880+0000", 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013289500000000001 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113897+0000", 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6762e-05 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.146839+0000", 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032941830999999998 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.1342365210000001, 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032588833999999997, 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.366 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184469+0000", 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:11.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186107+0000", 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016386580000000001 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186149+0000", 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1878999999999997e-05 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186192+0000", 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3130000000000002e-05 2024-09-21T15:17:11.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186334+0000", 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142235 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186359+0000", 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4260000000000002e-05 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.217059+0000", 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030700531999999999 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.369 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.0978649980000004, 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036223334000000003, 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220840+0000", 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222163+0000", 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001322939 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222190+0000", 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7030999999999999e-05 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222219+0000", 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8646e-05 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222352+0000", 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013276799999999999 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222375+0000", 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3467000000000001e-05 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.257065+0000", 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034689910999999997 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:11.373 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 8.055835149, 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:11.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:11.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.9816099329999997, 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:11.377 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:11.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:11.380 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.9712104940000001, 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.384 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.407304+0000", 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.9114026580000001, 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030614235, 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.407304+0000", 2024-09-21T15:17:11.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.407304+0000", 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.407303+0000", 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.408940+0000", 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016372839999999999 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.408975+0000", 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4714e-05 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409002+0000", 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7764999999999999e-05 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409101+0000", 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9152000000000002e-05 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409119+0000", 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7104000000000001e-05 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.437918+0000", 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028799874 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.908903488, 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:11.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:11.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:11.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.470145+0000", 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.848561568, 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032019508000000002, 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.392 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.470145+0000", 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.470145+0000", 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.470144+0000", 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471431+0000", 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00128681 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471454+0000", 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3244e-05 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471468+0000", 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4099e-05 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471543+0000", 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4993000000000005e-05 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.471560+0000", 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.715e-05 2024-09-21T15:17:11.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502165+0000", 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030604612 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 7.8466934200000003, 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.396 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.399 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.400 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.400 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:11.401 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19144 2024-09-21T15:17:11.401 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19144 2024-09-21T15:17:11.401 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19144 -C rbd-nbd 2024-09-21T15:17:11.401 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:11.401 INFO:tasks.workunit.client.0.smithi146.stdout: 19144 ? 00:00:00 rbd-nbd 2024-09-21T15:17:11.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:11.403 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:11.401+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:11.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=19144 2024-09-21T15:17:11.403 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd17425 2024-09-21T15:17:11.403 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:11.403 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:11.404 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:11.404 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:11.404 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:11.404 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:11.404 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd17425 2024-09-21T15:17:11.502 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:11.504+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:11.516 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:11.516 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:11.516 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:11.516 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:11.526 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:11.603 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:11.601+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:11.630 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:11.630 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:11.630 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:11.631 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:11.631 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:11.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:11.704 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:11.704+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:11.758 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.652838343999999, 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:11.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:11.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:11.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.762 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.652787301, 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.766 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.637167758, 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.767 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.637141005, 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.771 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:11.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.468092901999999, 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.775 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.452948044999999, 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.781 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.423801228999999, 2024-09-21T15:17:11.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:11.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:11.791 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.414474324, 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.792 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.797 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.386838172000001, 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:11.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.375760466999999, 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:11.803 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.808 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.340990919999999, 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:11.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:11.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:11.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.452466837999999, 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:11.814 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:11.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:11.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.819 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.442211771, 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:11.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.410164773999998, 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:11.825 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:11.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:11.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:11.830 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.397952821000001, 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.831 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:11.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:11.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:11.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.364056862999998, 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:11.836 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:11.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:11.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:11.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.325011994, 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:11.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:11.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.847 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.464568867000001, 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:11.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.45299136, 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:11.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:11.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.418580592, 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:11.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:11.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:11.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:11.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:11.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:11.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:11.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:11.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:11.865 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:11.867 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:11.865+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:11.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:11.868 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd1 2024-09-21T15:17:11.868 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd ns 2024-09-21T15:17:11.868 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns=ns 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19192 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19192 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19192 -C rbd-nbd 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stdout: 19144 ? 00:00:00 rbd-nbd 2024-09-21T15:17:11.869 INFO:tasks.workunit.client.0.smithi146.stdout: 19192 ? 00:00:00 rbd-nbd 2024-09-21T15:17:11.870 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device rbd/ns/testrbdnbd17425 19192 2024-09-21T15:17:11.870 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=rbd/ns/testrbdnbd17425 2024-09-21T15:17:11.870 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19192 2024-09-21T15:17:11.870 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd17425 2024-09-21T15:17:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:11.871 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd17425 2024-09-21T15:17:11.962 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:11.963 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19192\b' 2024-09-21T15:17:11.963 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19192\b' 2024-09-21T15:17:11.967 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:11.965+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:11.992 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:11.993 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:11.993 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:11.993 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:12.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:12.036 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:12.037 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:12.038 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19192\b' 2024-09-21T15:17:12.038 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19192\b' 2024-09-21T15:17:12.049 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:12.050 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:12.068 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:12.068+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:12.107 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:12.107 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:12.107 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:12.107 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:12.107 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:12.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:12.169 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:12.169+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:12.227 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.768353536999999, 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.228 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:12.233 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.72046928, 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.672297434000001, 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:12.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:12.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:12.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.314128331999999, 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:12.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:12.250 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.148144461000001, 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:12.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:12.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.962639192000001, 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.256 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:12.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.984000361, 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.262 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:12.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.983215483999999, 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:12.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:12.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.982675360000002, 2024-09-21T15:17:12.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:12.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:12.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.982472944000001, 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:12.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.982276739, 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.285 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:12.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:12.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.982082616, 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.291 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:12.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:12.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.296 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.927635394999999, 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:12.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:12.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:12.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.927502948000001, 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:12.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:12.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.927351568999999, 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:12.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.313 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.927224198000001, 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:12.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:12.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.792424197999999, 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:12.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:12.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:12.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:12.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:12.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.655398114, 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:12.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:12.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.480101210000001, 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:12.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:12.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.429808685000001, 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:12.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:12.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:12.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.342 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:12.343 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:12.343 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:12.343 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T15:17:12.343 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:17:12.343 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:17:12.343 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-21T15:17:12.345 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:12.341+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:12.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:12.346 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:17:12.444 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:12.445+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:12.460 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:12.460 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "" 2024-09-21T15:17:12.460 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:12.507 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:12.507 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:12.507 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:12.507 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:12.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:12.545 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:12.545+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:12.552 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:12.552 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device rbd/testrbdnbd17425 19144 2024-09-21T15:17:12.552 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=rbd/testrbdnbd17425 2024-09-21T15:17:12.552 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19144 2024-09-21T15:17:12.552 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd17425 2024-09-21T15:17:12.552 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:12.552 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:12.554 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:12.554 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:12.557 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:12.557 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:12.557 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd17425 2024-09-21T15:17:12.621 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:12.622 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:12.622 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:12.622 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:12.622 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:12.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:12.646 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:12.648+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:12.662 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:12.662 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19144\b' 2024-09-21T15:17:12.662 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19144\b' 2024-09-21T15:17:12.728 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:12.729 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:12.729 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19144\b' 2024-09-21T15:17:12.729 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19144\b' 2024-09-21T15:17:12.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:12.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:12.741 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:12.741 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 114.958686949, 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:12.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:12.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:624 2.7 e57/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 57'33, mlcod=57'31)", 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.320002862999999, 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029592388000000001, 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420905+0000", 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.420904+0000", 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421288+0000", 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000384674 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421318+0000", 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0017000000000001e-05 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421332+0000", 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3444e-05 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421441+0000", 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109409 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.421481+0000", 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0125e-05 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450452+0000", 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028971001 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.450498+0000", 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5037999999999998e-05 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.281749727000001, 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.953730358000001, 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:12.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:12.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:12.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.761 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.902568915, 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:12.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:12.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:12.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:12.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:12.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.814522949000001, 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:658 2.7 e57/56 2:f998d3cd:::rbd_data.13743413a48b.0000000000000007:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.594458551999999, 2024-09-21T15:17:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031302035999999998, 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146449+0000", 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146448+0000", 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146818+0000", 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036965199999999999 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146845+0000", 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7525000000000002e-05 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146858+0000", 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2703e-05 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146954+0000", 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6186000000000004e-05 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.146992+0000", 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7815999999999998e-05 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177692+0000", 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030700439999999999 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.177751+0000", 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9271000000000002e-05 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:674 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'7, mlcod=50'5)", 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.275393952000002, 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030213265, 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465514+0000", 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465512+0000", 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465905+0000", 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000392972 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465940+0000", 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5026e-05 2024-09-21T15:17:12.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.465954+0000", 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3256e-05 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466019+0000", 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5701e-05 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.466046+0000", 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6658000000000001e-05 2024-09-21T15:17:12.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495689+0000", 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029642966999999999 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.495727+0000", 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8352000000000001e-05 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.233991169999999, 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.784 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:688 2.5 e58/56 2:ae3b040b:::rbd_data.13743413a48b.000000000000000a:head v 58'61, mlcod=58'59)", 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.972755595999999, 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030671810000000001, 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:12.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768152+0000", 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768151+0000", 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768527+0000", 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037648899999999998 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768553+0000", 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6327999999999999e-05 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768565+0000", 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1467999999999999e-05 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768687+0000", 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121721 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.768746+0000", 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9200999999999998e-05 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798766+0000", 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030020013000000002 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.798824+0000", 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.827746487999999, 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.264962718, 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.793 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.798 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.182253944999999, 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:12.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:12.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:12.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:12.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.140134670999998, 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:12.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:12.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:12.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:12.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:57 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'112, mlcod=62'110)", 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.034258517000001, 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030174240000000001, 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706649+0000", 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.706648+0000", 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707026+0000", 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037793499999999998 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707044+0000", 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7445e-05 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707051+0000", 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7749999999999993e-06 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707107+0000", 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5374000000000002e-05 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.707130+0000", 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3283000000000001e-05 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736698+0000", 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029568046000000001 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.736824+0000", 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012566899999999999 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.99364417, 2024-09-21T15:17:12.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.897338747999999, 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:12.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:12.821 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.347168276, 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.826 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.294329751999999, 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.827 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:12.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.24848824, 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:12.832 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:12.837 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:12.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:12.839 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:12.840+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:12.939 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:12.940+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:13.000 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:13.000 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:13.000 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:13.000 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:13.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:13.040 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:13.041+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:13.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:13.138 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:13.138 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:13.141 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:13.140+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:13.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:13.240 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' rbd = rbd ']' 2024-09-21T15:17:13.241 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:13.241+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd17425 2024-09-21T15:17:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:13.241 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:13.242 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:13.243 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:13.244 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:13.246 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd17425 2024-09-21T15:17:13.263 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:13.283 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:13.307 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:13.312 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:13.312 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:13.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:13.330 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:13.355 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:13.368 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:13.369+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:13.407 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:13.408 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:13.408 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:13.408 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:13.408 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:13.409 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:13.442 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:13.444+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:13.450 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:13.450 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:13.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:13.497 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19289 2024-09-21T15:17:13.497 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19289 2024-09-21T15:17:13.497 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19289 -C rbd-nbd 2024-09-21T15:17:13.506 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:13.506 INFO:tasks.workunit.client.0.smithi146.stdout: 19289 ? 00:00:00 rbd-nbd 2024-09-21T15:17:13.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device testrbdnbd17425 19289 2024-09-21T15:17:13.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=testrbdnbd17425 2024-09-21T15:17:13.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19289 2024-09-21T15:17:13.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17425 2024-09-21T15:17:13.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:13.507 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:13.508 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:13.508 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:13.510 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:13.510 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:13.510 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17425 2024-09-21T15:17:13.543 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:13.541+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:13.569 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:13.569 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:13.579 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:13.620 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:13.621 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19289\b' 2024-09-21T15:17:13.621 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19289\b' 2024-09-21T15:17:13.644 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:13.644+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:13.686 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:13.686 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:13.687 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19289\b' 2024-09-21T15:17:13.687 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19289\b' 2024-09-21T15:17:13.692 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:13.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:13.732 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:13.732 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:13.743 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:13.744 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:13.745+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:13.845 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:13.845 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:13.846 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:13.845+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:13.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:13.946 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:13.948+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:14.008 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.008 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.018 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:14.047 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:14.045+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:14.121 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.121 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.131 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:14.147 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:14.148+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:14.194 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:14.194 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd config global set global rbd_default_pool new_default_pool17425 2024-09-21T15:17:14.239 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.239 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.248 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:14.245+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:14.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:14.253 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd create --size 10M testrbdnbd17425 2024-09-21T15:17:14.329 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd17425 2024-09-21T15:17:14.329 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:14.329 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:14.331 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:14.332 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:14.334 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:14.334 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:14.334 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd17425 2024-09-21T15:17:14.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:17:14.349 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:14.348+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:14.350 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.350 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:14.450 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:14.449+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:14.466 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.466 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:14.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:14.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid new_default_pool17425 2024-09-21T15:17:14.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=new_default_pool17425 2024-09-21T15:17:14.507 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:14.508 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:14.508 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool17425'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:14.550 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:14.549+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:14.589 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.589 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19417 2024-09-21T15:17:14.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19417 2024-09-21T15:17:14.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19417 -C rbd-nbd 2024-09-21T15:17:14.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:14.603 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:14.604 INFO:tasks.workunit.client.0.smithi146.stdout: 19417 ? 00:00:00 rbd-nbd 2024-09-21T15:17:14.604 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device testrbdnbd17425 19417 2024-09-21T15:17:14.604 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=testrbdnbd17425 2024-09-21T15:17:14.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19417 2024-09-21T15:17:14.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd17425 2024-09-21T15:17:14.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:14.605 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:14.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:14.606 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:14.608 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:14.608 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:14.608 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd17425 2024-09-21T15:17:14.651 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:14.652+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:14.705 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.705 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.714 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:14.714 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19417\b' 2024-09-21T15:17:14.714 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19417\b' 2024-09-21T15:17:14.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:14.752 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:14.752+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:14.780 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:14.781 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:14.782 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19417\b' 2024-09-21T15:17:14.782 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19417\b' 2024-09-21T15:17:14.791 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:14.792 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:14.819 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.819 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:14.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:14.872+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:14.954 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:14.953+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:14.982 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:14.982 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:14.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:15.055 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:15.053+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:15.103 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:15.103 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:15.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:15.158 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:15.160+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:15.221 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:15.221 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:15.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:15.255 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:15.256+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:15.265 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:17:15.293 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:15.293 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd config global rm global rbd_default_pool 2024-09-21T15:17:15.345 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:15.345 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:15.355 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:17:15.355 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:15.355 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:15.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:15.357 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:15.357+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:15.357 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:15.358 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:15.360 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:15.360 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:15.360 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:17:15.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:15.457 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:15.456+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:15.458 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:15.458 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:15.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:15.514 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:15.515 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:15.515 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:15.515 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:15.515 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:15.516 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:15.541 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:15.558 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:15.557+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:15.568 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:15.575 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:15.575 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:15.585 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:15.587 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:15.587 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19513 2024-09-21T15:17:15.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19513 2024-09-21T15:17:15.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19513 -C rbd-nbd 2024-09-21T15:17:15.600 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:15.600 INFO:tasks.workunit.client.0.smithi146.stdout: 19513 ? 00:00:00 rbd-nbd 2024-09-21T15:17:15.600 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo kill 19513 2024-09-21T15:17:15.600 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:15.600 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:15.602 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:15.602 INFO:tasks.workunit.client.0.smithi146.stderr:++ which kill 2024-09-21T15:17:15.604 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/kill 2024-09-21T15:17:15.604 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:15.604 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/kill 19513 2024-09-21T15:17:15.609 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:15.626 INFO:tasks.workunit.client.0.smithi146.stderr:++ seq 10 2024-09-21T15:17:15.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ for i in `seq 10` 2024-09-21T15:17:15.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:15.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19513 *rbd *testrbdnbd17425' 2024-09-21T15:17:15.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19513 *rbd *testrbdnbd17425' 2024-09-21T15:17:15.632 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:15.654 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:15.658 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:15.657+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:15.703 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:15.704 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:15.704 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:15.704 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:15.711 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:15.711 INFO:tasks.workunit.client.0.smithi146.stderr:+ break 2024-09-21T15:17:15.712 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19513 *rbd *testrbdnbd17425' 2024-09-21T15:17:15.712 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19513 *rbd *testrbdnbd17425' 2024-09-21T15:17:15.712 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:15.714 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:15.759 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:15.760+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:15.787 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:15.787 INFO:tasks.workunit.client.0.smithi146.stderr:+ QUIESCE_HOOK=/tmp/tmp.flUinkcHzA/quiesce.sh 2024-09-21T15:17:15.788 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.flUinkcHzA/quiesce.sh rbd/testrbdnbd17425 2024-09-21T15:17:15.788 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:15.788 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:15.789 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:15.789 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:15.791 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:15.791 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:15.791 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.flUinkcHzA/quiesce.sh rbd/testrbdnbd17425 2024-09-21T15:17:15.821 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:15.821 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:15.821 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:15.821 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:15.821 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:15.830 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:15.859 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:15.857+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.318573499999999, 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.945 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:15.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:15.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:15.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:15.950 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.278126907999997, 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:15.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:15.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.954 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.202425562999998, 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:15.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:15.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:15.958 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.155805166, 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:15.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:15.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.962 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.105188542, 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:15.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:15.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:15.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:15.966 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.016835069999999, 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:15.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.970 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.527704199999999, 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:15.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:15.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:15.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.812823357999999, 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.768116532000001, 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.979 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:15.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.724509100999999, 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:15.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:15.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.628200524, 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:15.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:15.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.467091125, 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:15.991 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:15.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:15.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:15.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.421760655, 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:15.995 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:15.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.342537272000001, 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:15.999 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.509064278, 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.456898823, 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:16.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:16.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.412107912, 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:16.011 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:16.015 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 21.372584843999999, 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:16.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.019 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.639447266000001, 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.023 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.605163493999999, 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:16.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:16.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:16.028 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.028 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.028 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.028 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.028 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.028 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:16.029 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:16.030 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:16.030 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:16.030 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:16.030 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:16.030 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:16.030 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:16.031 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:16.028+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:16.031 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19577 2024-09-21T15:17:16.031 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19577 2024-09-21T15:17:16.031 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19577 -C rbd-nbd 2024-09-21T15:17:16.043 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:16.043 INFO:tasks.workunit.client.0.smithi146.stdout: 19577 ? 00:00:00 rbd-nbd 2024-09-21T15:17:16.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ test '!' -e /tmp/tmp.flUinkcHzA/quiesce.sh 2024-09-21T15:17:16.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false rbd snap create rbd/testrbdnbd17425@quiesce1 2024-09-21T15:17:16.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd snap create rbd/testrbdnbd17425@quiesce1 2024-09-21T15:17:16.121 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete...2024-09-21T15:17:16.118+0000 7f0a7affd700 -1 librbd::ImageWatcher: 0x7f0a5c0075a0 failed to notify quiesce: (1) Operation not permitted 2024-09-21T15:17:16.121 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:17:16.118+0000 7f0a7affd700 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-09-21T15:17:16.123 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete...failed. 2024-09-21T15:17:16.124 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-09-21T15:17:16.130 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:16.129+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:16.140 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:16.140 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:16.140 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:16.141 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:16.143 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:16.143 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:17:16.146 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:17:16.146 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:16.146 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:16.150 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:16.150 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:16.150 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:16.150 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:16.160 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:17:16.221 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-09-21T15:17:16.222 INFO:tasks.workunit.client.0.smithi146.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685951 s, 15.3 MB/s 2024-09-21T15:17:16.223 INFO:tasks.workunit.client.0.smithi146.stderr:+ touch /tmp/tmp.flUinkcHzA/quiesce.sh 2024-09-21T15:17:16.225 INFO:tasks.workunit.client.0.smithi146.stderr:+ chmod +x /tmp/tmp.flUinkcHzA/quiesce.sh 2024-09-21T15:17:16.226 INFO:tasks.workunit.client.0.smithi146.stderr:+ cat 2024-09-21T15:17:16.229 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd snap create rbd/testrbdnbd17425@quiesce1 2024-09-21T15:17:16.231 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:16.232+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:16.282 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:16.282 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:16.283 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:16.283 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:16.283 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:16.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:16.336 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:16.337+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:ac59e578:ns::rbd_data.137aae78387e.0000000000000007:head, version: 50'15, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ac59e578:ns::rbd_data.137aae78387e.0000000000000007:head@50'15, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:16.395 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 14.053340972000001, 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.03268339, 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339331+0000", 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.339330+0000", 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340709+0000", 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013791540000000001 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340735+0000", 2024-09-21T15:17:16.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5687e-05 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340755+0000", 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0428000000000002e-05 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340873+0000", 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011763 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.340890+0000", 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6713000000000001e-05 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.372014+0000", 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031124496000000001 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:b3267759:ns::rbd_data.137aae78387e.000000000000000a:head, version: 50'20, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:b3267759:ns::rbd_data.137aae78387e.000000000000000a:head@50'20, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:16.399 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.924515725999999, 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033844346999999997, 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468156+0000", 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.468154+0000", 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469522+0000", 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0013677920000000001 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469560+0000", 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8232999999999998e-05 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469584+0000", 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3595999999999999e-05 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469777+0000", 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000192871 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.469814+0000", 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7174e-05 2024-09-21T15:17:16.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.502000+0000", 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032186335000000003 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head, version: 63'92, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head@63'92, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.797578165999999, 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032502650000000001, 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.403 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.595093+0000", 2024-09-21T15:17:16.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596191+0000", 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010984600000000001 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596207+0000", 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5761999999999999e-05 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596216+0000", 2024-09-21T15:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2410000000000001e-06 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596392+0000", 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017578099999999999 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.596413+0000", 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1267000000000002e-05 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.627596+0000", 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031182781999999999 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.5 67/66 [PushOp(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head, version: 63'92, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a2f32205:::rbd_data.13743413a48b.0000000000000006:head@63'92, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.703435925000001, 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035308071000000003, 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689236+0000", 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.689234+0000", 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690873+0000", 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016385620000000001 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690906+0000", 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2920000000000003e-05 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.690930+0000", 2024-09-21T15:17:16.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4391000000000001e-05 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.691115+0000", 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000184554 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.691153+0000", 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8371999999999998e-05 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:02.724544+0000", 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033390729000000001 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head, version: 62'100, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6473f039:::rbd_data.13743413a48b.0000000000000001:head@62'100, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.280099276, 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034266194, 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.112572+0000", 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113722+0000", 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011501860000000001 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113737+0000", 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5199e-05 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113747+0000", 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9650000000000003e-06 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113880+0000", 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013289500000000001 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.113897+0000", 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6762e-05 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.146839+0000", 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032941830999999998 2024-09-21T15:17:16.414 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.208201244, 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032588833999999997, 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184470+0000", 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.184469+0000", 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186107+0000", 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0016386580000000001 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186149+0000", 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1878999999999997e-05 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186192+0000", 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3130000000000002e-05 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186334+0000", 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142235 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.186359+0000", 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4260000000000002e-05 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.217059+0000", 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030700531999999999 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.418 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.171829721, 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036223334000000003, 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:16.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220842+0000", 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.220840+0000", 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222163+0000", 2024-09-21T15:17:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001322939 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222190+0000", 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7030999999999999e-05 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222219+0000", 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8646e-05 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222352+0000", 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013276799999999999 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.222375+0000", 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3467000000000001e-05 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.257065+0000", 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034689910999999997 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.129799872, 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:16.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.426 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.055574655999999, 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:16.430 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 13.045175217000001, 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 12.982868211, 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.434 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:16.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:16.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:16.437 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 12.920658143000001, 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:16.441 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.22352290699999999, 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.442 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:16.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.447 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.223072929, 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.22282306399999999, 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.453 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:16.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:16.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.22254756000000001, 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:16.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:16.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:16.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.222144595, 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.464 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.22193855900000001, 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.470 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:16.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:16.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.475 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.221734556, 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:16.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:16.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.221543674, 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.481 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:16.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:16.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:16.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.487 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:16.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:16.488 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:16.488+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:16.588 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:16.589+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:16.601 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:16.601 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:16.601 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:16.601 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:16.611 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:16.689 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:16.688+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:16.764 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:16.765 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:16.765 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:16.765 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:16.765 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:16.774 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:16.790 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:16.792+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:16.884 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:16.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.885 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:16.885 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:16.885 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.780025561999999, 2024-09-21T15:17:16.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:16.885 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.885 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.885 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:16.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:16.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:16.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:16.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:16.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:16.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.779974519, 2024-09-21T15:17:16.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.893 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:16.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.897 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.764354976, 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.898 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:16.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:16.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:16.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.764328223, 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:16.902 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.595280119999998, 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:16.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:16.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:16.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.580135263000003, 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.913 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:16.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:16.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.550988447000002, 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.919 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:16.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:16.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:16.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.541661542, 2024-09-21T15:17:16.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:16.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:16.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:16.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:16.929 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.51402539, 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:16.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:16.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.502947685000002, 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:16.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.941 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.468178137999999, 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:16.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:16.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:16.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:16.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.579654055999999, 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:16.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.949 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:16.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:16.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:16.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.569398989, 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:16.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:16.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.960 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.537351992000001, 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:16.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:16.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:16.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:16.966 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.966 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.966 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.966 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.966 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:16.967 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.525140039, 2024-09-21T15:17:16.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:16.967 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.967 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.967 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.967 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.967 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.968 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:16.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.968 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:16.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:16.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:16.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:16.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:16.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:16.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:16.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.491244081000001, 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.976 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:16.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.977 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:16.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:16.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:16.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:16.982 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.452199212, 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:16.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:16.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:16.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.591756085, 2024-09-21T15:17:16.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:16.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:16.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:16.994 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.580178578000002, 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:16.995 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:16.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:16.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:17.000 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 19.545767810000001, 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.001 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:17.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:17.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:17.006 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:17.007 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:17.007 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:17.007 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:17.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:17.009 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:17.008+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:17.109 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:17.108+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:17.125 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:17.125 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:17.125 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:17.126 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:17.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:17.211 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:17.209+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:17.245 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:17.245 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:17.245 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:17.245 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:17.246 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:17.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:17.274 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T15:17:17.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:17:17.285 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:17.285 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:17.286 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:17.288 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:17.288 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:17:17.290 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:17:17.291 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:17.291 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:17.312 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:17.309+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.906872223000001, 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:17.365 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:17.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:17.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.858987966000001, 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:17.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.810816119999998, 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:17.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:17.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:17.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.452647018, 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:17.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:17.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.286663146999999, 2024-09-21T15:17:17.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:17.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:17.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.101157878000002, 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.394 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:17.400 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.122519047000001, 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.12173417, 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:17.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:17.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.121194045999999, 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:17.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:17.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.120991629999999, 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:17.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:17.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:17.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:17.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.120795425000001, 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:17.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.120601302000001, 2024-09-21T15:17:17.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.066154081000001, 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:17.433 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.434 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.434 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.434 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.434 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.434 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:17.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.434 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:17.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:17.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:17.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.066021633999998, 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:17.440 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:17.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:17.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.446 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.065870255, 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:17.447 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:17.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.065742883999999, 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:17.454 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:17.459 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.930942884, 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:17.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:17.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:17.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.793916800000002, 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:17.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.618619895999998, 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:17.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:17.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.568327370999999, 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:17.477 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:17.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.483 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:17.484 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:17.484 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:17.484 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T15:17:17.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:17:17.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:17.487 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:17.484+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:17:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-09-21T15:17:17.488 INFO:tasks.workunit.client.0.smithi146.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676411 s, 15.5 MB/s 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:++ cat /tmp/tmp.flUinkcHzA/quiesce 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:++ cat /tmp/tmp.flUinkcHzA/unquiesce 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ touch /tmp/tmp.flUinkcHzA/quiesce.sh 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ chmod +x /tmp/tmp.flUinkcHzA/quiesce.sh 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ cat 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false rbd snap create rbd/testrbdnbd17425@quiesce2 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd snap create rbd/testrbdnbd17425@quiesce2 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:17.489 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:17:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:17:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:17.490 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:17.543 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:17:17.543 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-09-21T15:17:17.543 INFO:tasks.workunit.client.0.smithi146.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554823 s, 18.9 MB/s 2024-09-21T15:17:17.544 INFO:tasks.workunit.client.0.smithi146.stderr:+ cat 2024-09-21T15:17:17.546 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd snap create rbd/testrbdnbd17425@quiesce2 2024-09-21T15:17:17.586 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:17.585+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:17.600 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:17.600 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:17.600 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:17.600 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:17.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:17.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:17.687 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:17.685+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:17.727 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:17.727 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:17.727 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:17.727 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:17.727 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:17.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:17.788 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:17.789+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:17.808 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:17.828 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 77, 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:17:17.248466+0000", 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:17:00.287583+0000", 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5804032, 2024-09-21T15:17:17.846 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-09-21T15:17:17.847 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:17.848 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:17:17.850 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:17.852 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:17:17.853 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:17.854 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:17.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "77", 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:17.856 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 6, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 77, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:17.857 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:17.858 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.25, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.25, 2024-09-21T15:17:17.859 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2.25, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2.25, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "new_default_pool17425", 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:16:34.640599+0000", 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:17.860 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "54", 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 54, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:17.864 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:17.865 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.2699999809265137, 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.2699999809265137, 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.85597997903823853, 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.866 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 68, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:17.867 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:17.868 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:17.869 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:17.870 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:17:17.871 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:17.872 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.427032470703125, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:17:17.874 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:17:17.875 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:17:17.876 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:17.877 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 72, 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.879 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:17:17.881 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:17.887 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:17:17.888 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:17.889 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 69, 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:17:17.892 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:17.894 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:17.895 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:17.900 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:17.901 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:17.903 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:17:17.904 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.1", 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: 4, 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: 5 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.1", 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:17.905 INFO:teuthology.orchestra.run.smithi067.stdout: 2, 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: 3 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [], 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:17:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:17:17.907 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:17:17.908 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:17:17.908 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:17.908 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:17.908 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:17.910 INFO:tasks.thrashosds.thrasher:j is {'epoch': 77, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:17:17.248466+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:17:00.287583+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 77, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 3, 'pool_name': 'new_default_pool17425', 'create_time': '2024-09-21T15:16:34.640599+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8559799790382385, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 68, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.427032470703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.1', 'osds': [4, 5]}, {'pgid': '3.1', 'osds': [2, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:17:17.910 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T15:17:17.911 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:17.911 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:17.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:17.912 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:17.912 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:17.912 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:17.909+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:17.915 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:18.011 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:18.012+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:18.017 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.017 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:18.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:18.111 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:18.112+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:18.181 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.181 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:18.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:18.211 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:18.212+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:18.273 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:17:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.303 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:18.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:18.313 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:18.313+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:18.414 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:18.413+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:18.430 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.430 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:18.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:18.515 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:18.513+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:18.547 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.547 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:18.557 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:18.616 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:18.613+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:18.709 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.709 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:18.717 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:18.717+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:18.719 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:18.818 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:18.817+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:18.881 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.881 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:18.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:18.919 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:18.917+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:18.996 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:18.996 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.006 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:19.048 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:19.049+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:19.121 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:19.119+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:19.158 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:19.158 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:19.222 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:19.221+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:19.275 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:19.275 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.285 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:19.323 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:19.323+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:19.424 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:19.425+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:19.438 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:19.438 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:19.525 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:19.523+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:19.552 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:19.552 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:19.625 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:19.625+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:19.679 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:19.679 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:19.726 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:19.727+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:19.793 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:19.793 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:19.827 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:19.827+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:19.912 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:19.912 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:19.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:19.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:19.966 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:19.967+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:20.028 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:20.029+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:20.074 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:20.074 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:20.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:20.107 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:20.129 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:20.127+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:20.134 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:20.154 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:20.178 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:20.184 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:20.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:20.184 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:20.184 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:20.184 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:20.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:20.201 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:20.230 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:20.229+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:20.230 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:20.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:17:20.321 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 122.539010785, 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.862073563000003, 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.534054193999999, 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.482892751000001, 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.394846784999999, 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.814315006000001, 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.408070324000001, 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.845286553999998, 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.762577780999997, 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.720458507000004, 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.573968006000001, 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.477662584000001, 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.927492111999999, 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.874653588000001, 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.828812075999998, 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:98 2.5 e76/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 76'106, mlcod=72'103)", 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.149110673, 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033104706999999997, 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172355+0000", 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023398099999999999 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172384+0000", 2024-09-21T15:17:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.917e-05 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172407+0000", 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2739e-05 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172480+0000", 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3712000000000002e-05 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172516+0000", 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6208999999999998e-05 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.205119+0000", 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032602617 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.205226+0000", 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106805 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.9967219109999998, 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.9931603490000001, 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.9893751750000002, 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.8143403330000001, 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.426 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:20.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:20.428 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:20.425+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:20.528 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:20.527+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:20.536 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:20.536 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:20.536 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:20.536 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:20.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:20.629 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:20.627+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:20.697 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:20.697 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:20.697 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:20.697 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:20.697 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:20.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:20.730 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:20.729+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:20.821 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.99880653, 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:20.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.436382028000001, 2024-09-21T15:17:20.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:20.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:20.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.425645527, 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:20.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:20.835 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.381188692999999, 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.836 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.373411891, 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.33423822, 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:20.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.327505858999999, 2024-09-21T15:17:20.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:20.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.293454366999999, 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:20.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:20.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.283328383000001, 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:20.861 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:20.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.586564999, 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:20.866 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.449185696000001, 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:20.871 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:20.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.448361644, 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:20.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:20.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.563300096000001, 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:20.880 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:20.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:20.884 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.562344025, 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:20.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:20.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.889 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.423249943, 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029330900000000001, 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397916+0000", 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:20.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397928+0000", 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1897e-05 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397948+0000", 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0319999999999999e-05 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397964+0000", 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5868000000000001e-05 2024-09-21T15:17:20.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398041+0000", 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6278000000000004e-05 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398130+0000", 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9227999999999993e-05 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398212+0000", 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1964000000000003e-05 2024-09-21T15:17:20.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.422668642, 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:20.894 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.898 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1294768990000001, 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000394242, 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.899 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691689+0000", 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691697+0000", 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.174e-06 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691708+0000", 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0492000000000001e-05 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691716+0000", 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5399999999999996e-06 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691813+0000", 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6644000000000005e-05 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691957+0000", 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014423300000000001 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692086+0000", 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012863100000000001 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1287553670000001, 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:20.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:20.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:20.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:20.907 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:20.908 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:20.909 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:20.909 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:20.907+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:21.010 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:21.009+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:21.028 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:21.028 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:21.028 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:21.029 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:21.038 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:21.111 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:21.111+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:21.146 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:21.147 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:21.147 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:21.147 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:21.147 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:21.156 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:21.212 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:21.208+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:21.267 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:21.267 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:21.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.640896167000001, 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:21.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:21.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:21.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.600449574999999, 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:21.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.52474823, 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:21.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:21.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:21.280 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.478127833000002, 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:21.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:21.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.427511209000002, 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:21.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.339157737000001, 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.850026866999997, 2024-09-21T15:17:21.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:21.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:21.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.135146024999997, 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.301 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.090439199000002, 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:21.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:21.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:21.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.046831767999997, 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:21.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.950523191000002, 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:21.314 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.789413791999998, 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:21.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.744083322000002, 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.319 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:21.323 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 35.664859939000003, 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:21.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.327 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.831386944999998, 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:21.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:21.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:21.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.779221490000001, 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.734430579000001, 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 26.694907511, 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.961769932999999, 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.927486161000001, 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:21.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:21.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.353 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:17:21.355 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:21.355 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:21.352+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:21.456 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:21.455+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:21.465 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:21.465 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:21.466 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:21.466 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:21.475 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:21.557 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:21.555+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:21.581 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:21.582 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:21.582 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:21.591 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:21.658 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:21.656+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:21.705 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.705 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.439492833999999, 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:21.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.709 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.365267618000001, 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.713 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.354868179, 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.714 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.715 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.716 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.292561172999999, 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:21.719 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:21.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.721 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.230351105, 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:21.722 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:21.723 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.724 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5332158690000002, 2024-09-21T15:17:21.725 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.726 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.727 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.728 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.729 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.730 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5327658910000004, 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.731 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:21.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:21.734 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:21.735 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5325160259999997, 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:21.736 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.737 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.738 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.739 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:21.740 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:21.741 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5322405220000004, 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.742 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.743 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.744 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.745 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.746 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.747 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5318375570000002, 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:21.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:21.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:21.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:21.752 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5316315209999996, 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:21.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:21.758 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.5314275180000001, 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:21.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:21.763 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.531236636, 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:21.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:21.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:21.769 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.3941733100000002, 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:17:21.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:17:21.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:17:21.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:21.775 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.3939382880000002, 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:17:21.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:17:21.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.3937378650000003, 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:17:21.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:17:21.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:17:21.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.3933342829999997, 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.3930751260000003, 2024-09-21T15:17:21.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:17:21.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:17:21.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:17:21.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.3928798990000004, 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.798 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:21.803 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 4.1986281349999999, 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:17:21.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:21.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:21.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:17:21.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:17:21.809 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.809 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.809 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.809 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:21.809 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:21.809 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:21.810 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:21.811 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:21.811+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:21.912 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:21.911+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:21.963 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:21.963 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:21.963 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:21.963 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:21.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:22.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:22.011+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:22.076 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:22.076 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:22.076 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:22.076 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:22.076 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:22.086 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:22.112 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:22.112+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:22.192 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:22.192 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:22.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:22.192 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:22.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.192 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:22.192 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.086874211999998, 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:22.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:22.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.086823168999999, 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.197 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.071203625999999, 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.071176872999999, 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.902128769999997, 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.886983913000002, 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:22.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.857837097000001, 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.220 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:22.225 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.848510191999999, 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:22.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.82087404, 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:22.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:22.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:22.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.235 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.809796335000001, 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.236 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:22.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:22.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.775026787999998, 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:22.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:22.246 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.886502706000002, 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:22.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:22.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:22.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.251 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.876247638999999, 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:22.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:22.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:22.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.844200641999997, 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.257 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:22.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.831988688999999, 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.262 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:22.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:22.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.798092730999997, 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.759047862000003, 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.898604734999999, 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:22.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.887027228000001, 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.284 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 24.85261646, 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:22.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:22.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:22.295 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:22.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:22.297 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:22.295+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:22.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:22.397 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:22.395+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:22.411 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:22.411 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:22.411 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:22.411 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:22.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:22.445 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:22.466 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:22.484 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:22.498 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:22.499+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:22.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:22.517 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:22.517 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:22.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:22.532 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:22.555 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:22.599 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:22.596+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:22.638 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:22.638 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:22.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:22.700 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:22.699+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:22.754 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:22.755 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:22.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:22.801 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:22.799+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:22.873 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:22.873 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:22.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:22.901 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:22.899+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:22.911 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:22.911 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:22.911 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T15:17:22.911 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-21T15:17:22.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-21T15:17:22.994 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:22.994 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.003 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:23.003+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:23.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:23.103 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:23.100+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:23.105 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.105 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:23.204 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:23.200+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:23.218 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.218 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T15:17:23.305 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:23.304+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:23.335 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.335 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.345 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:23.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:17:23.407 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:23.404+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:23.459 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.459 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.469 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:23.507 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:23.507+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:23.573 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.573 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.582 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:23.608 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:23.607+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:23.686 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.686 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:23.740 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:23.740+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:23.804 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.804 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.810 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:23.807+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:23.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:23.910 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:23.911+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:23.924 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:23.924 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:23.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:24.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:24.011+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:24.040 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:24.040 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:24.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:24.112 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:24.111+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:24.155 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:24.155 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:24.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:24.214 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:24.211+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:24.272 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:24.272 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:24.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:24.303 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:17:24.315 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:24.312+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:24.383 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:24.383 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:24.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:24.416 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:24.416+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:24.506 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:24.507 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:24.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:24.517 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:24.516+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:24.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:24.617 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:24.616+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:24.626 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:24.627 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:24.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:24.705 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:24.718 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:24.719+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:24.734 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.289544454000001, 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.241660197000002, 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.193488350999999, 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.835319249000001, 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.669335378, 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.483830109000003, 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.505191278000002, 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.504406401000001, 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:24.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:24.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.503866277, 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:24.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:24.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.503663861, 2024-09-21T15:17:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.503467656000002, 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.503273533000002, 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.448826312000001, 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.448693864999999, 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.448542486000001, 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.448415115, 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.313615115000001, 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.176589030999999, 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.001292126999999, 2024-09-21T15:17:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.950999602, 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:24.860 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:24.861 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:24.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:24.863 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:24.863 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:24.863 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:24.863 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:24.864 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:24.864 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:24.864 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:24.864 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:24.864 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:24.864 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:24.859+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:24.963 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:24.960+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:25.021 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:25.021 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:25.021 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:25.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:25.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:25.063+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:25.141 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:25.141 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:25.141 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:25.142 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:25.142 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:25.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:25.165 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:25.164+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:25.264 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 127.481047597, 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:25.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:25.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.804110375, 2024-09-21T15:17:25.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:25.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.476091005999997, 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:25.281 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.424929562999999, 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:25.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.336883597000003, 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:25.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:25.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:25.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.756351817999999, 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:25.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.350107135999998, 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:25.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:25.301 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.787323366000003, 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:25.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:25.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.704614593000002, 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:25.307 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:25.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:25.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:25.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.662495319000001, 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:25.313 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:25.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:25.318 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.516004817999999, 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:25.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.322 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.419699395999999, 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:25.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:25.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:25.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.869528924000001, 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:25.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:25.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.816690399999999, 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.332 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:25.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.337 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.770848888, 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:25.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:25.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:98 2.5 e76/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 76'106, mlcod=72'103)", 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.0911474850000005, 2024-09-21T15:17:25.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033104706999999997, 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172121+0000", 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172355+0000", 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023398099999999999 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172384+0000", 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.917e-05 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172407+0000", 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2739e-05 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172480+0000", 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3712000000000002e-05 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.172516+0000", 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6208999999999998e-05 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.205119+0000", 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032602617 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:16.205226+0000", 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106805 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:25.347 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.9387587230000003, 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.351 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.9351971609999996, 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:25.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.9314119869999997, 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:25.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.7563771450000001, 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.360 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:25.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:25.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:25.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.364 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:25.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:25.367 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:25.363+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:25.367 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-09-21T15:17:25.367 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:25.367 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:25.367 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:25.367 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:25.367 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:17:25.368 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:17:25.368 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:25.368 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-09-21T15:17:25.428 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:17:25.428 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-09-21T15:17:25.428 INFO:tasks.workunit.client.0.smithi146.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575882 s, 18.2 MB/s 2024-09-21T15:17:25.430 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 19577 2024-09-21T15:17:25.430 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:17:25.430 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19577 2024-09-21T15:17:25.430 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:17:25.430 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:25.430 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:25.432 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:25.433 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:25.435 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:25.435 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:25.436 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:17:25.466 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:25.467+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:25.525 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:25.526 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:25.526 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:25.526 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:25.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:25.567 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:25.567+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:25.639 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:25.639 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:25.639 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:25.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:25.668 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:25.668+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:25.769 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:25.768+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:25.808 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.808 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 122.98440317799999, 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.421978675999998, 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.813 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:25.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.818 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.411242175000002, 2024-09-21T15:17:25.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:25.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:25.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:25.827 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.366785341, 2024-09-21T15:17:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:25.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:25.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:25.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:25.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:25.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.836 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.359008539000001, 2024-09-21T15:17:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:25.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:25.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.846 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:25.846 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:25.846 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.319834868000001, 2024-09-21T15:17:25.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:25.847 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.847 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.847 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:25.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:25.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:25.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:25.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:25.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:25.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:25.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.313102507, 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:25.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:25.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.279051015, 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:25.858 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.862 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.268925030999998, 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:25.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:25.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.572161647000001, 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:25.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:25.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:25.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:25.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.434782343999998, 2024-09-21T15:17:25.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:25.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.433958292, 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.877 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:25.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:25.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.548896744, 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:25.882 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:25.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.547940672999999, 2024-09-21T15:17:25.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:25.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.408846591, 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029330900000000001, 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.891 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397916+0000", 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397928+0000", 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1897e-05 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397948+0000", 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0319999999999999e-05 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397964+0000", 2024-09-21T15:17:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5868000000000001e-05 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398041+0000", 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6278000000000004e-05 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398130+0000", 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9227999999999993e-05 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398212+0000", 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1964000000000003e-05 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 17.408265289999999, 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.895 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:25.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:25.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:25.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:25.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:25.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.115073547, 2024-09-21T15:17:25.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000394242, 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691689+0000", 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691697+0000", 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.174e-06 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691708+0000", 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0492000000000001e-05 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691716+0000", 2024-09-21T15:17:25.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5399999999999996e-06 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691813+0000", 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6644000000000005e-05 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691957+0000", 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014423300000000001 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692086+0000", 2024-09-21T15:17:25.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012863100000000001 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.114352015, 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.905 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:25.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:25.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.47441327500000002, 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:17:25.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:25.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:17:25.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.5557.0:362 2.0 e81/62 ondisk, result = 0)", 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.46967181899999999, 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031135899999999998, 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:25.916 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:25.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337092+0000", 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337101+0000", 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6670000000000002e-06 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337120+0000", 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9623000000000002e-05 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337134+0000", 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3842000000000001e-05 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337214+0000", 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9992000000000003e-05 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337230+0000", 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6246999999999999e-05 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:25.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337405+0000", 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174193 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:25.920 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:25.922 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:25.922 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:25.919+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:26.023 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:26.023+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:26.040 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:26.040 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:26.040 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:26.040 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:26.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:26.123 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:26.120+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:26.161 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:26.161 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:26.161 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:26.161 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:26.161 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:26.170 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:26.224 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:26.223+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:26.281 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:26.281 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:26.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:26.281 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:26.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.281 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:26.281 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.654501494999998, 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:26.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.614054903000003, 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.286 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:26.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.538353557999997, 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:26.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:26.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.491733160999999, 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:26.295 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:26.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:26.299 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.441116536999999, 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:26.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.352763064999998, 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:26.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.863632195000001, 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:26.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:26.313 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.148751353000002, 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:26.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.104044526999999, 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:26.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:26.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.060437096000001, 2024-09-21T15:17:26.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:26.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.326 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.964128518999999, 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:26.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:26.331 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.803019120000002, 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:26.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.335 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.757688649999999, 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:26.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:26.340 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 40.678465267, 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:26.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.844992272999999, 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:26.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:26.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.792826818000002, 2024-09-21T15:17:26.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:26.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:26.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.748035906999998, 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:26.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.708512839000001, 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:26.358 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:26.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:26.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.975375261, 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:26.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.941091489000002, 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:26.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:26.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.372 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.372 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:26.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:17:26.374 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:26.374 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:26.372+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:26.475 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:26.472+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:26.489 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:26.489 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:26.489 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:26.489 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:26.499 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:26.575 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:26.572+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:26.649 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:26.649 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:26.650 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:26.650 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:26.650 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:26.659 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:26.704 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:26.704+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:26.770 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:26.770 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:26.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:26.770 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:26.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.771 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:26.771 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:26.771 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.503780567, 2024-09-21T15:17:26.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:26.771 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.771 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:26.772 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:26.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:26.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:26.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.429555351000001, 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:26.780 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.419155912000001, 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:26.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:26.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:26.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:26.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:26.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:26.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:26.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:26.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:26.787 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.787 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.787 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.787 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:26.787 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:26.788 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.356848906, 2024-09-21T15:17:26.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:26.788 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.788 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:26.788 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:26.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:26.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:26.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:26.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:26.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:26.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:26.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:26.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:26.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:26.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:26.795 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.795 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.795 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.796 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:26.796 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:26.796 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.294638838000001, 2024-09-21T15:17:26.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:26.796 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.796 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:26.797 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:26.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:26.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:26.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:26.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:26.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.597503602, 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:26.803 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:26.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.597053624000001, 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.809 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:26.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:26.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.814 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.596803759, 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.815 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:26.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:26.820 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.596528254999999, 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:26.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:26.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:26.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.59612529, 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:26.827 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.832 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.595919254, 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.833 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:26.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:26.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.595715251, 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:26.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:26.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:26.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.595524369, 2024-09-21T15:17:26.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:26.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:26.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:26.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:26.850 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.4584610429999998, 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:17:26.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:17:26.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.4582260209999998, 2024-09-21T15:17:26.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:17:26.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:17:26.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.4580255980000008, 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.862 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:26.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:17:26.867 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.4576220160000002, 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.868 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:17:26.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.4573628589999998, 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:26.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:17:26.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:26.879 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.4571676320000009, 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:26.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:17:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 9.2629158680000003, 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:17:26.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:17:26.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:26.891 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:26.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:26.893 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:26.891+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:26.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:26.994 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:26.991+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:27.006 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:27.006 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:27.006 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:27.006 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:27.016 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:27.045 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:27.068 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:27.091 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:27.094 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:27.095+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:27.114 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:27.125 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:27.126 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:27.126 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:27.126 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:27.126 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:27.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:27.138 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:27.158 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:27.195 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:27.192+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:27.245 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:27.245 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:27.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:27.296 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:27.295+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:27.349 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:17:27.376 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:27.376 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:27.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:27.398 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:27.396+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:27.496 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:27.496 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:27.498 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:27.499+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:27.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:27.599 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:27.596+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:27.660 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:27.661 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:27.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:27.700 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:27.699+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:27.775 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:27.775 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:27.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:27.800 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:27.800+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:27.901 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:27.900+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:27.936 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:27.936 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:27.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:28.002 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:28.000+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:28.050 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.051 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:28.106 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:28.107+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:28.169 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.169 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:28.203 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:28.200+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:28.288 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.288 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:28.304 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:28.303+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:28.312 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:28.312 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:28.312 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T15:17:28.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:17:28.405 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:28.404+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:28.408 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.408 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:28.506 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:28.503+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:28.532 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.532 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:28.606 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:28.607+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:28.612 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:28.612 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:28.612 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 83, 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:17:27.329876+0000", 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:17:00.287583+0000", 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5804032, 2024-09-21T15:17:28.613 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:17:28.614 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:17:28.615 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-09-21T15:17:28.615 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:17:28.615 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:17:28.615 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-09-21T15:17:28.615 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:17:28.615 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:17:28.615 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:17:28.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.616 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:17:28.616 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:17:28.616 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:17:28.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:17:28.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:28.617 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:28.618 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:28.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:28.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:28.619 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:28.619 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:28.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.620 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:17:28.620 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:28.620 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:28.620 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:28.620 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:28.620 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:28.620 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:28.621 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:28.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:28.623 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:28.624 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:17:28.625 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:28.626 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:28.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "82", 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 7, 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 82, 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:28.628 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:28.629 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:28.630 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.25, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.25, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2.25, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2.25, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:28.631 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "new_default_pool17425", 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:16:34.640599+0000", 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:28.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:28.633 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "54", 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:28.634 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 54, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:28.635 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:28.636 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:28.637 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.2699999809265137, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.2699999809265137, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.638 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 68, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:28.639 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.640 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.642 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 80, 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.646 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:17:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:17:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.648 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.648 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:28.648 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.648 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:28.649 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:17:28.651 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:17:28.652 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:28.653 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:28.653 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:28.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.653 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:28.653 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:28.654 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 72, 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:17:28.655 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:17:28.656 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:17:28.657 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.658 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:28.659 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:28.660 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:17:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:28.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 69, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:28.667 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.669 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.670 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:28.672 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:17:28.677 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:28.680 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:28.681 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.682 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:28.683 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.1", 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: 4, 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: 5 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:28.684 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.1", 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: 2, 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: 3 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [], 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:17:28.685 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:17:28.686 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:17:28.687 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:17:28.687 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:17:28.687 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:17:28.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:28.687 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:28.688 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.688 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.689 INFO:tasks.thrashosds.thrasher:j is {'epoch': 83, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:17:27.329876+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:17:00.287583+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 82, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'new_default_pool17425', 'create_time': '2024-09-21T15:16:34.640599+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 68, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.1', 'osds': [4, 5]}, {'pgid': '3.1', 'osds': [2, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:17:28.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:17:28.690 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:28.707 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:28.704+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:28.801 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.802 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.808 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:28.807+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:28.812 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:28.909 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:28.907+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:28.923 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:28.923 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:28.932 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:29.005 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:29.007 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:17:29.009 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:29.007+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:29.019 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-21T15:17:27.951136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":137954135,"num_objects":58,"num_object_clones":3,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":4452,"num_read_kb":150466,"num_write":755,"num_write_kb":268858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":79691903,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":716,"ondisk_log_size":716,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":459720,"kb_used_data":273012,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":565771320,"statfs":{"total":579820584960,"available":579349831680,"internally_reserved":0,"allocated":279564288,"data_stored":292753436,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1048576,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":525,"num_write":5,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.659572"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":227,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T15:17:24.297297+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:17:24.297297+0000","last_peered":"2024-09-21T15:17:24.297297+0000","last_clean":"2024-09-21T15:17:24.297297+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:17:24.297297+0000","last_undegraded":"2024-09-21T15:17:24.297297+0000","last_fullsized":"2024-09-21T15:17:24.297297+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"72'40","reported_seq":950,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-21T15:17:25.483045+0000","last_change":"2024-09-21T15:16:55.672969+0000","last_active":"2024-09-21T15:17:25.483045+0000","last_peered":"2024-09-21T15:17:25.483045+0000","last_clean":"2024-09-21T15:17:25.483045+0000","last_became_active":"2024-09-21T15:16:55.272135+0000","last_became_peered":"2024-09-21T15:16:55.272135+0000","last_unstale":"2024-09-21T15:17:25.483045+0000","last_undegraded":"2024-09-21T15:17:25.483045+0000","last_fullsized":"2024-09-21T15:17:25.483045+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:58.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":16777254,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":791,"num_read_kb":29143,"num_write":39,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"75'6","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.296153+0000","last_change":"2024-09-21T15:16:55.572914+0000","last_active":"2024-09-21T15:17:23.296153+0000","last_peered":"2024-09-21T15:17:23.296153+0000","last_clean":"2024-09-21T15:17:23.296153+0000","last_became_active":"2024-09-21T15:16:55.273190+0000","last_became_peered":"2024-09-21T15:16:55.273190+0000","last_unstale":"2024-09-21T15:17:23.296153+0000","last_undegraded":"2024-09-21T15:17:23.296153+0000","last_fullsized":"2024-09-21T15:17:23.296153+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:38.164122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181755,"stat_sum":{"num_bytes":54,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"50'4","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.296329+0000","last_change":"2024-09-21T15:16:55.569580+0000","last_active":"2024-09-21T15:17:23.296329+0000","last_peered":"2024-09-21T15:17:23.296329+0000","last_clean":"2024-09-21T15:17:23.296329+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:17:23.296329+0000","last_undegraded":"2024-09-21T15:17:23.296329+0000","last_fullsized":"2024-09-21T15:17:23.296329+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388624,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":3,"num_write_kb":8193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.296478+0000","last_change":"2024-09-21T15:17:08.376120+0000","last_active":"2024-09-21T15:17:23.296478+0000","last_peered":"2024-09-21T15:17:23.296478+0000","last_clean":"2024-09-21T15:17:23.296478+0000","last_became_active":"2024-09-21T15:17:08.375700+0000","last_became_peered":"2024-09-21T15:17:08.375700+0000","last_unstale":"2024-09-21T15:17:23.296478+0000","last_undegraded":"2024-09-21T15:17:23.296478+0000","last_fullsized":"2024-09-21T15:17:23.296478+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:49:16.960044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"64'20","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.294409+0000","last_change":"2024-09-21T15:17:01.739071+0000","last_active":"2024-09-21T15:17:23.294409+0000","last_peered":"2024-09-21T15:17:23.294409+0000","last_clean":"2024-09-21T15:17:23.294409+0000","last_became_active":"2024-09-21T15:17:01.316522+0000","last_became_peered":"2024-09-21T15:17:01.316522+0000","last_unstale":"2024-09-21T15:17:23.294409+0000","last_undegraded":"2024-09-21T15:17:23.294409+0000","last_fullsized":"2024-09-21T15:17:23.294409+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:15.523790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":16384,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T15:17:25.182368+0000","last_change":"2024-09-21T15:16:55.272512+0000","last_active":"2024-09-21T15:17:25.182368+0000","last_peered":"2024-09-21T15:17:25.182368+0000","last_clean":"2024-09-21T15:17:25.182368+0000","last_became_active":"2024-09-21T15:16:55.271874+0000","last_became_peered":"2024-09-21T15:16:55.271874+0000","last_unstale":"2024-09-21T15:17:25.182368+0000","last_undegraded":"2024-09-21T15:17:25.182368+0000","last_fullsized":"2024-09-21T15:17:25.182368+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:54:48.789474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046520300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"63'22","reported_seq":244,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-21T15:17:25.348293+0000","last_change":"2024-09-21T15:17:01.768225+0000","last_active":"2024-09-21T15:17:25.348293+0000","last_peered":"2024-09-21T15:17:25.348293+0000","last_clean":"2024-09-21T15:17:25.348293+0000","last_became_active":"2024-09-21T15:17:01.327838+0000","last_became_peered":"2024-09-21T15:17:01.327838+0000","last_unstale":"2024-09-21T15:17:25.348293+0000","last_undegraded":"2024-09-21T15:17:25.348293+0000","last_fullsized":"2024-09-21T15:17:25.348293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:08.047427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":111,"num_read_kb":8285,"num_write":22,"num_write_kb":20482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'1","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T15:17:24.301212+0000","last_change":"2024-09-21T15:16:55.676403+0000","last_active":"2024-09-21T15:17:24.301212+0000","last_peered":"2024-09-21T15:17:24.301212+0000","last_clean":"2024-09-21T15:17:24.301212+0000","last_became_active":"2024-09-21T15:16:55.274451+0000","last_became_peered":"2024-09-21T15:16:55.274451+0000","last_unstale":"2024-09-21T15:17:24.301212+0000","last_undegraded":"2024-09-21T15:17:24.301212+0000","last_fullsized":"2024-09-21T15:17:24.301212+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:20.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046455,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"75'1","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.296149+0000","last_change":"2024-09-21T15:16:55.271559+0000","last_active":"2024-09-21T15:17:23.296149+0000","last_peered":"2024-09-21T15:17:23.296149+0000","last_clean":"2024-09-21T15:17:23.296149+0000","last_became_active":"2024-09-21T15:16:55.271108+0000","last_became_peered":"2024-09-21T15:16:55.271108+0000","last_unstale":"2024-09-21T15:17:23.296149+0000","last_undegraded":"2024-09-21T15:17:23.296149+0000","last_fullsized":"2024-09-21T15:17:23.296149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:30.189660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"63'25","reported_seq":339,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T15:17:25.182425+0000","last_change":"2024-09-21T15:15:25.427839+0000","last_active":"2024-09-21T15:17:25.182425+0000","last_peered":"2024-09-21T15:17:25.182425+0000","last_clean":"2024-09-21T15:17:25.182425+0000","last_became_active":"2024-09-21T15:15:22.822789+0000","last_became_peered":"2024-09-21T15:15:22.822789+0000","last_unstale":"2024-09-21T15:17:25.182425+0000","last_undegraded":"2024-09-21T15:17:25.182425+0000","last_fullsized":"2024-09-21T15:17:25.182425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:41.716380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":25165824,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":242,"num_read_kb":11153,"num_write":25,"num_write_kb":32770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"75'1","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.294424+0000","last_change":"2024-09-21T15:17:01.318184+0000","last_active":"2024-09-21T15:17:23.294424+0000","last_peered":"2024-09-21T15:17:23.294424+0000","last_clean":"2024-09-21T15:17:23.294424+0000","last_became_active":"2024-09-21T15:17:01.317913+0000","last_became_peered":"2024-09-21T15:17:01.317913+0000","last_unstale":"2024-09-21T15:17:23.294424+0000","last_undegraded":"2024-09-21T15:17:23.294424+0000","last_fullsized":"2024-09-21T15:17:23.294424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:47:53.254684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043867299999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"82'127","reported_seq":1077,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-21T15:17:25.475425+0000","last_change":"2024-09-21T15:17:24.302020+0000","last_active":"2024-09-21T15:17:25.475425+0000","last_peered":"2024-09-21T15:17:25.475425+0000","last_clean":"2024-09-21T15:17:25.475425+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:17:25.475425+0000","last_undegraded":"2024-09-21T15:17:25.475425+0000","last_fullsized":"2024-09-21T15:17:25.475425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":24117302,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1035,"num_read_kb":34486,"num_write":140,"num_write_kb":49188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.294499+0000","last_change":"2024-09-21T15:17:01.321005+0000","last_active":"2024-09-21T15:17:23.294499+0000","last_peered":"2024-09-21T15:17:23.294499+0000","last_clean":"2024-09-21T15:17:23.294499+0000","last_became_active":"2024-09-21T15:17:01.320694+0000","last_became_peered":"2024-09-21T15:17:01.320694+0000","last_unstale":"2024-09-21T15:17:23.294499+0000","last_undegraded":"2024-09-21T15:17:23.294499+0000","last_fullsized":"2024-09-21T15:17:23.294499+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:22.824314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092366200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"77'186","reported_seq":974,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-21T15:17:24.631590+0000","last_change":"2024-09-21T15:17:05.211552+0000","last_active":"2024-09-21T15:17:24.631590+0000","last_peered":"2024-09-21T15:17:24.631590+0000","last_clean":"2024-09-21T15:17:24.631590+0000","last_became_active":"2024-09-21T15:17:05.209676+0000","last_became_peered":"2024-09-21T15:17:05.209676+0000","last_unstale":"2024-09-21T15:17:24.631590+0000","last_undegraded":"2024-09-21T15:17:24.631590+0000","last_fullsized":"2024-09-21T15:17:24.631590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:02:52.206876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777254,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1580,"num_read_kb":34110,"num_write":108,"num_write_kb":49165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":20,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T15:17:23.294472+0000","last_change":"2024-09-21T15:17:01.320994+0000","last_active":"2024-09-21T15:17:23.294472+0000","last_peered":"2024-09-21T15:17:23.294472+0000","last_clean":"2024-09-21T15:17:23.294472+0000","last_became_active":"2024-09-21T15:17:01.320707+0000","last_became_peered":"2024-09-21T15:17:01.320707+0000","last_unstale":"2024-09-21T15:17:23.294472+0000","last_undegraded":"2024-09-21T15:17:23.294472+0000","last_fullsized":"2024-09-21T15:17:23.294472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:29:15.250976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"82'166","reported_seq":407,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-21T15:17:25.340367+0000","last_change":"2024-09-21T15:17:24.303933+0000","last_active":"2024-09-21T15:17:25.340367+0000","last_peered":"2024-09-21T15:17:25.340367+0000","last_clean":"2024-09-21T15:17:25.340367+0000","last_became_active":"2024-09-21T15:17:24.303534+0000","last_became_peered":"2024-09-21T15:17:24.303534+0000","last_unstale":"2024-09-21T15:17:25.340367+0000","last_undegraded":"2024-09-21T15:17:25.340367+0000","last_fullsized":"2024-09-21T15:17:25.340367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:39.035665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":20971596,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":478,"num_read_kb":16767,"num_write":233,"num_write_kb":53389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":137363697,"num_objects":50,"num_object_clones":3,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":4298,"num_read_kb":150336,"num_write":590,"num_write_kb":266436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":79691884,"num_keys_recovered":20,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":273747968,"data_stored":289407496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77172,"kb_used_data":46048,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94294668,"statfs":{"total":96636764160,"available":96557740032,"internally_reserved":0,"allocated":47153152,"data_stored":47090389,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.66300000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.133,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.878}]},{"osd":2,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.032,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.82699999999999996,"15min":0.83699999999999997},"min":{"1min":0.53100000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2070000000000001,"5min":1.212,"15min":1.212},"last":0.79100000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65300000000000002,"15min":0.64500000000000002},"min":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84699999999999998,"15min":0.84999999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0880000000000001,"5min":1.208,"15min":1.208},"last":0.83899999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.82399999999999995,"15min":0.83599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0089999999999999,"5min":1.629,"15min":1.629},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.76400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56636,"kb_used_data":25516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315204,"statfs":{"total":96636764160,"available":96578768896,"internally_reserved":0,"allocated":26128384,"data_stored":25528517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.99099999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.83099999999999996},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96199999999999997,"5min":1.333,"15min":1.333},"last":0.59699999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.115,"5min":1.133,"15min":1.133},"last":0.72399999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.64200000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.84299999999999997,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.038,"5min":1.581,"15min":1.581},"last":0.75800000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.626}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81299999999999994,"15min":0.81499999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.86599999999999999,"15min":0.86499999999999999},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.49399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92536,"kb_used_data":61420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94279304,"statfs":{"total":96636764160,"available":96542007296,"internally_reserved":0,"allocated":62894080,"data_stored":72256327,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.42899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.84299999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.90300000000000002,"5min":1.226,"15min":1.226},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.751}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84699999999999998,"15min":0.84599999999999997},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.272,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86499999999999999,"15min":0.85999999999999999},"min":{"1min":0.64900000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":1.2589999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.81699999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81212,"kb_used_data":50096,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94290628,"statfs":{"total":96636764160,"available":96553603072,"internally_reserved":0,"allocated":51298304,"data_stored":50694404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57799999999999996,"15min":0.59199999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.81599999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.88300000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95499999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.84999999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91300000000000003,"5min":1.101,"15min":1.101},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.90100000000000002,"15min":0.91500000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.085,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73699999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.90400000000000003,"15min":0.92200000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0289999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.91900000000000004,"15min":0.92900000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1839999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.0089999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607583,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95536,"kb_used_data":64420,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94276304,"statfs":{"total":96636764160,"available":96538935296,"internally_reserved":0,"allocated":65966080,"data_stored":71660565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.76100000000000001,"15min":0.77300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90200000000000002,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.76400000000000001,"15min":0.78100000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.331,"15min":1.331},"last":0.93000000000000005}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.82899999999999996,"15min":0.84699999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.86899999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.80500000000000005,"15min":0.81100000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.101,"5min":1.298,"15min":1.298},"last":0.622}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.04,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78100000000000003,"15min":0.78700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93500000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.84499999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82599999999999996,"15min":0.83699999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.048,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.79900000000000004}]}]},{"osd":2,"up_from":12,"seq":51539607581,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56628,"kb_used_data":25512,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315212,"statfs":{"total":96636764160,"available":96578777088,"internally_reserved":0,"allocated":26124288,"data_stored":25523234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":1.052}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.70099999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97399999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.82999999999999996,"15min":0.83799999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":1.091},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.123}]},{"osd":4,"last update":"Sat Sep 21 15:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.093,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":1.0089999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95999999999999996,"15min":0.96799999999999997},"min":{"1min":0.622,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.94799999999999995,"15min":0.95499999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.88600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50348032,"data_stored":50331781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61886464,"data_stored":71303336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25165878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65028096,"data_stored":71303279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":46145536,"data_stored":46137398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:17:29.020 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [5, 4, 0, 3] 2024-09-21T15:17:29.020 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.4', '5', '4', '0', '3'] 2024-09-21T15:17:29.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.4 5 4 0 3 2024-09-21T15:17:29.036 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:29.036 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:29.046 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:29.110 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:29.111+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:29.148 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:29.148 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:29.159 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:29.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:29.212 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:29.211+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:29.264 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:29.264 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:29.274 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:29.312 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:29.312+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:29.317 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:29.343 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:29.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:29.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:17:29.377 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:29.377 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:29.387 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:29.388 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:29.410 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:29.413 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:29.411+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:29.432 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:29.514 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:29.515+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:29.550 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:29.550 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:29.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:29.551 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:29.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.551 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:29.551 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:29.552 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.444561252, 2024-09-21T15:17:29.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:29.553 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.553 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:29.553 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.553 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.553 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.554 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:29.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.554 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:29.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:29.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:29.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:29.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:29.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:29.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:29.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:29.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:29.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:29.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:29.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:29.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:29.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:29.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:29.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:29.563 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.564 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.564 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.564 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:29.565 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:29.565 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.444510209000001, 2024-09-21T15:17:29.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:29.565 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.565 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:29.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.566 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:29.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.567 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:29.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:29.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:29.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:29.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:29.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:29.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:29.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:29.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:29.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:29.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:29.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:29.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:29.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:29.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:29.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:29.575 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.575 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.575 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.576 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:29.576 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:29.576 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.428890666000001, 2024-09-21T15:17:29.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:29.576 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.577 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:29.577 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.577 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.577 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.577 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:29.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.577 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:29.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:29.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:29.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:29.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:29.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:29.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.428863913000001, 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.587 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.259815809999999, 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.588 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:29.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:29.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:29.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:29.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:29.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.244670953000004, 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.594 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:29.599 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.215524137000003, 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:29.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:29.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:29.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:29.604 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.206197232000001, 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:29.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:29.610 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.178561080000001, 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:29.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:29.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:29.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.167483375000003, 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.616 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.132713828, 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:29.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:29.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:29.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:29.626 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.244189746000004, 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:29.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:29.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:29.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.233934679000001, 2024-09-21T15:17:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.201887681999999, 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.189675729000001, 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.155779770999999, 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:29.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:29.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:29.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:29.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.116734901999997, 2024-09-21T15:17:29.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:29.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:29.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.256291775000001, 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:29.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:29.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:29.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:29.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:29.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:29.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.244714268000003, 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:29.666 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.210303500000002, 2024-09-21T15:17:29.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.676 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:29.677 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:29.677 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:29.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:29.678 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:29.679+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:29.778 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:29.779+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:29.790 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:29.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:29.790 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:29.790 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:29.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:29.880 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:29.880+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:29.952 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:29.952 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:29.952 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:29.952 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:29.952 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:29.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:29.981 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:29.980+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:30.071 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.611789725000001, 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:30.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.563905468000002, 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.077 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:30.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:30.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:30.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.515733621999999, 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:30.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.157564520000001, 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:30.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.094 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.991580648999999, 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:30.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:30.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:30.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.100 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.806075380000003, 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.827436548999998, 2024-09-21T15:17:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.826651671999997, 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.826111548, 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.825909132, 2024-09-21T15:17:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.825712926999998, 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.825518803999998, 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.771071583000001, 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.770939136000003, 2024-09-21T15:17:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.770787757000001, 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.770660386000003, 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.635860385999997, 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.498834301999999, 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.323537397999999, 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.273244873000003, 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.186 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:30.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:30.188 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:30.188+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:30.288 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:30.288+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:30.301 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:30.301 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:30.301 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:30.301 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:30.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:30.363 INFO:teuthology.orchestra.run.smithi067.stderr:set 2.4 pg_upmap_items mapping to [5->4,0->3] 2024-09-21T15:17:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:17:30.390 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:30.388+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:30.421 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:30.421 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:30.421 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:30.421 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:30.422 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:30.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:30.491 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:30.488+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:30.538 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:30.538 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:30.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:30.538 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:30.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 132.755172168, 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:30.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.078234946000002, 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.545 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.750215576999999, 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:30.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.699054134000001, 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.561 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.611008167999998, 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:30.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.030476389, 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:30.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.624231707, 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:30.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:30.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.061447936999997, 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.978739163999997, 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.583 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:30.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:30.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.936619890000003, 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:30.589 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:30.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.594 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.790129389000001, 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:30.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.693823967, 2024-09-21T15:17:30.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:30.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.143653495000002, 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:30.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.090814971, 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:30.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.044973458999998, 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:30.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.212883293999999, 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.621 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:30.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.209321731999999, 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:30.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:30.629 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.205536558, 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.030501716, 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.634 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:30.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1427093360000002, 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:17:30.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:17:30.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:30.644 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:30.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:30.646 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:30.647+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:30.746 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:30.744+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:30.808 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:30.808 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:30.808 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:30.808 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:30.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:30.847 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:30.844+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:30.927 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:30.928 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:30.928 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:30.928 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:30.928 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:30.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:30.948 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:30.944+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:31.047 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:31.047 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:31.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:31.047 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.22463088000001, 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:31.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.662206378, 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:31.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:31.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.651469876999997, 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.607013043000002, 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:31.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:31.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.599236241, 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.072 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.560062569999999, 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:31.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.553330209000002, 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.077 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:31.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.519278716999999, 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:31.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.509152733000001, 2024-09-21T15:17:31.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:31.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.812389349, 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:31.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:31.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.675010046000001, 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:31.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:31.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:31.100 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.674185993999998, 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:31.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.789124445999999, 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:31.105 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.109 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.788168375000001, 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:31.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.649074292999998, 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029330900000000001, 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.114 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397916+0000", 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397928+0000", 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1897e-05 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397948+0000", 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0319999999999999e-05 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397964+0000", 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5868000000000001e-05 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398041+0000", 2024-09-21T15:17:31.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6278000000000004e-05 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398130+0000", 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9227999999999993e-05 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398212+0000", 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1964000000000003e-05 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.648492992000001, 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:31.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.355301249, 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000394242, 2024-09-21T15:17:31.123 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691689+0000", 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691697+0000", 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.174e-06 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691708+0000", 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0492000000000001e-05 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691716+0000", 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5399999999999996e-06 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691813+0000", 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6644000000000005e-05 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691957+0000", 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014423300000000001 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692086+0000", 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012863100000000001 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.127 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.354579717, 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:31.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:31.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.7146409770000002, 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:31.132 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:31.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:17:31.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:17:31.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.5557.0:362 2.0 e81/62 ondisk, result = 0)", 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.7098995209999996, 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031135899999999998, 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:31.139 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:31.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337092+0000", 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337101+0000", 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6670000000000002e-06 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337120+0000", 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9623000000000002e-05 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337134+0000", 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3842000000000001e-05 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337214+0000", 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9992000000000003e-05 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337230+0000", 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6246999999999999e-05 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:31.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337405+0000", 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174193 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:31.143 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:31.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:31.145 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:31.144+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:31.245 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:31.247+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:31.260 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:31.327 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:31.328 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:31.329 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:31.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:31.348 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:31.347+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:31.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:31.447 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:31.447 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:31.447 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:31.447 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:31.447 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:31.448 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:31.447+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:31.457 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:31.550 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:31.548+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.939348758000001, 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.567 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:31.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:31.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:31.571 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.898902165999999, 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:31.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:31.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.575 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.823200821, 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:31.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:31.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:31.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.776580424000002, 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:31.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:31.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:31.584 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.725963800000002, 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:31.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.588 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.637610328000001, 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:31.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:31.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.148479457999997, 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:31.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:31.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.433598615999998, 2024-09-21T15:17:31.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:31.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.388891790000002, 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:31.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.345284358999997, 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.606 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.610 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.248975782000002, 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:31.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.087866382999998, 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.615 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:31.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:31.619 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.042535913000002, 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:31.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.623 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.963312530000003, 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:31.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:31.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.129839535999999, 2024-09-21T15:17:31.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:31.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.077674080999998, 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:31.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.032883169999998, 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:31.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:31.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.993360101999997, 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:31.641 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:31.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 34.260222524, 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:31.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 34.225938751999998, 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:31.650 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:31.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:31.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:31.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:31.654 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:31.656 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:31.656 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:31.656 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:31.657 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:31.657 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:31.652+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:31.667 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:31.688 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:31.711 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:31.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:31.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:31.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:31.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:31.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:31.714 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:31.757 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:31.756+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:31.819 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:31.819 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:31.829 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:31.858 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:31.859+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:31.934 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:31.934 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:31.945 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:31.958 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:31.956+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:32.057 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.058 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.059 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:32.059+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:32.067 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:32.160 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:32.159+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:32.214 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.215 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.225 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:32.261 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:32.260+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:32.330 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.330 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:32.361 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:32.359+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:32.374 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:32.374 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19577\b' 2024-09-21T15:17:32.374 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19577\b' 2024-09-21T15:17:32.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:17:32.444 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.444 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.445 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:32.446 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:32.446 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19577\b' 2024-09-21T15:17:32.446 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19577\b' 2024-09-21T15:17:32.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:32.459 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:32.459 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:32.463 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:32.460+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:32.563 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:32.563+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.566 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:32.664 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:32.663+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:32.680 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.680 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:32.766 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:32.763+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:32.793 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.793 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:32.867 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:32.864+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:32.909 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:32.910 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:32.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:32.961 INFO:tasks.workunit.client.0.smithi146.stderr:+ LOG_FILE=/tmp/tmp.flUinkcHzA/rbd-nbd.log 2024-09-21T15:17:32.961 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -n '' ']' 2024-09-21T15:17:32.962 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd17425 --log-file=/tmp/tmp.flUinkcHzA/rbd-nbd.log 2024-09-21T15:17:32.962 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:32.962 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:32.964 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:32.965 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:32.967 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:32.964+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:32.967 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:32.968 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:32.968 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd17425 --log-file=/tmp/tmp.flUinkcHzA/rbd-nbd.log 2024-09-21T15:17:33.025 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:33.025 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:33.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:33.068 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:33.067+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:33.134 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd0 2024-09-21T15:17:33.134 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:33.134 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:33.134 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:33.134 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:33.135 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:33.148 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:33.149 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:33.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:33.202 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:33.203+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:33.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19763 2024-09-21T15:17:33.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19763 2024-09-21T15:17:33.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19763 -C rbd-nbd 2024-09-21T15:17:33.220 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:33.220 INFO:tasks.workunit.client.0.smithi146.stdout: 19763 ? 00:00:00 rbd-nbd 2024-09-21T15:17:33.220 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo mkfs /dev/nbd0 2024-09-21T15:17:33.220 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:33.221 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:33.222 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:33.223 INFO:tasks.workunit.client.0.smithi146.stderr:++ which mkfs 2024-09-21T15:17:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/mkfs 2024-09-21T15:17:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:33.226 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd0 2024-09-21T15:17:33.265 INFO:tasks.workunit.client.0.smithi146.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-21T15:17:33.270 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:33.268+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:33.323 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:33.323 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:33.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:33.338 INFO:tasks.workunit.client.0.smithi146.stdout:Discarding device blocks: 4096/16384 done 2024-09-21T15:17:33.338 INFO:tasks.workunit.client.0.smithi146.stdout:Creating filesystem with 16384 4k blocks and 16384 inodes 2024-09-21T15:17:33.338 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-21T15:17:33.338 INFO:tasks.workunit.client.0.smithi146.stdout:Allocating group tables: 0/1 done 2024-09-21T15:17:33.352 INFO:tasks.workunit.client.0.smithi146.stdout:Writing inode tables: 0/1 done 2024-09-21T15:17:33.370 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:33.368+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:17:33.392 INFO:tasks.workunit.client.0.smithi146.stdout:Writing superblocks and filesystem accounting information: 0/1 done 2024-09-21T15:17:33.392 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-21T15:17:33.393 INFO:tasks.workunit.client.0.smithi146.stderr:+ mkdir /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:33.396 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:33.396 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:33.397 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:33.399 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:33.399 INFO:tasks.workunit.client.0.smithi146.stderr:++ which mount 2024-09-21T15:17:33.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/mount 2024-09-21T15:17:33.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:33.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:33.447 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd snap create rbd/testrbdnbd17425@quiesce3 2024-09-21T15:17:33.449 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:33.449 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:33.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:33.471 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:33.468+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:33.565 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:33.566 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:33.572 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:33.571+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:33.575 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:33.673 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:33.672+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:33.695 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:33.695 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:33.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:33.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:33.774 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:33.772+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:33.811 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:33.811 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:33.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:33.875 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:33.875+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:33.880 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:33.902 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:33.925 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:33.949 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:33.974 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:33.976 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:33.976+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:33.978 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:33.978 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.351999614, 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:33.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:33.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.311553021999998, 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:33.983 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.987 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.235851676999999, 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:33.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.991 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.189231280000001, 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:33.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:33.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:33.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.138614656000001, 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:33.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:33.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:33.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:33.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.050261184, 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.000 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:34.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:34.004 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.561130314000003, 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:34.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.008 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.846249471999997, 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:34.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.801542646000001, 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:34.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:34.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.757935215000003, 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:34.017 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.661626638000001, 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.025 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.500517238999997, 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:34.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:34.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:34.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.455186769000001, 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 48.375963386000002, 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.034 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:34.038 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.542490391999998, 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:34.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:34.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.042 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.490324936999997, 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:34.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:34.046 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.445534025999997, 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:34.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.406010958000003, 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.051 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:34.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:34.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.672873379999999, 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.055 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.056 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:34.059 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.638589607999997, 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.060 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:34.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:34.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.063 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.064 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:34.064 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:34.065 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:34.076 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:34.075+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:34.177 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:34.176+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:34.183 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:34.183 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:34.183 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:34.183 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:34.193 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:34.278 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:34.279+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:34.298 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:34.298 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:34.298 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:34.298 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:34.299 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:34.308 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:34.378 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:34.379+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:34.410 INFO:tasks.workunit.client.0.smithi146.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T15:17:34.419 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.154485139999998, 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:34.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.080259924, 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:34.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:34.427 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.069860485, 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:34.431 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 31.007553478999998, 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.433 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.434 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.435 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 30.945343411, 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:34.437 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.438 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.439 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.248208174999998, 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.440 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.441 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.442 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.443 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.445 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.247758197, 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.446 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.447 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.448 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.449 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.247508331999999, 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:34.451 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:34.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:34.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:34.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.247232828000001, 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:34.457 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.246829862999999, 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.246623827000001, 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.474 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.246419824, 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.475 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:34.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:34.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:34.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:34.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.246228941999998, 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:34.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:34.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.109165615999999, 2024-09-21T15:17:34.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:17:34.491 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.108930594, 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:17:34.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:17:34.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.108730171000001, 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:17:34.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.503 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.108326589000001, 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:17:34.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:17:34.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.108067431999999, 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:17:34.509 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:17:34.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.107872205, 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.515 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:17:34.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 16.913620440999999, 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.521 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:34.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:17:34.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:17:34.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.527 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:34.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:34.529 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:34.528+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:34.531 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/tmp/tmp.flUinkcHzA/mnt/test bs=1M count=1 oflag=direct 2024-09-21T15:17:34.531 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:34.531 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:34.531 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:34.531 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:17:34.531 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:17:34.531 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/tmp/tmp.flUinkcHzA/mnt/test bs=1M count=1 oflag=direct 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033696 s, 311 MB/s 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo umount /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:++ which umount 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/umount 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 19763 2024-09-21T15:17:34.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=19763 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:34.533 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:17:34.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:34.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^19763\b' 2024-09-21T15:17:34.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^19763\b' 2024-09-21T15:17:34.629 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:34.627+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:34.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:34.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:34.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *19763\b' 2024-09-21T15:17:34.673 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *19763\b' 2024-09-21T15:17:34.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:34.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:34.687 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:34.687 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:34.687 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:34.687 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:34.697 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:34.730 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:34.731+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:34.803 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:34.804 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:34.804 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:34.804 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:34.804 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:34.813 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:34.831 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:34.831+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:34.928 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:34.928 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:34.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:34.928 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.822529564999996, 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:34.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:34.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.822478521999997, 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.933 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:34.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:34.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.937 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.806858978999998, 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.941 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.806832225999997, 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.942 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:34.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:34.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.637784123000003, 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.946 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:34.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:34.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:34.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:34.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:34.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:34.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:34.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.622639266, 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:34.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.959 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.593492449999999, 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:34.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:34.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:34.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:34.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:34.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.966 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.584165544999998, 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.967 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:34.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:34.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:34.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:34.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:34.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:34.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:34.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:34.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:34.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:34.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:34.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:34.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:34.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:34.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:34.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:34.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:34.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:34.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:34.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.976 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.976 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.976 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.976 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:34.976 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.556529392999998, 2024-09-21T15:17:34.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:34.977 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.977 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.977 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.977 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.977 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.977 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:34.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.978 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:34.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:34.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:34.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:34.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:34.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:34.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:34.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.985 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.545451688, 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:34.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:34.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:34.991 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.510682140999997, 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.992 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:34.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:34.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:34.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:34.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.622158059, 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:34.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:34.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:35.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.611902991999997, 2024-09-21T15:17:35.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.008 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.579855995000003, 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:35.009 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:35.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:35.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:35.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.567644041999998, 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.533748084000003, 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.020 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:35.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:35.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.025 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.494703215000001, 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:35.026 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:35.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.634260087999998, 2024-09-21T15:17:35.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:35.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:35.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:35.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:35.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.038 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.622682580999999, 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.039 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 37.588271812999999, 2024-09-21T15:17:35.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:35.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:35.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:35.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:35.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:35.049 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.050 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:35.050 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:35.050 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:35.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:35.052 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:35.051+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:35.152 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:35.152+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:35.170 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:35.170 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:35.170 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:35.170 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:35.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:35.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:35.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ cat /tmp/tmp.flUinkcHzA/rbd-nbd.log 2024-09-21T15:17:35.182 INFO:tasks.workunit.client.0.smithi146.stdout:2024-09-21T15:17:33.058+0000 7f7e288763c0 0 ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process rbd-nbd, pid 19754 2024-09-21T15:17:35.182 INFO:tasks.workunit.client.0.smithi146.stdout:2024-09-21T15:17:33.062+0000 7f7e288763c0 0 pidfile_write: ignore empty --pid-file 2024-09-21T15:17:35.182 INFO:tasks.workunit.client.0.smithi146.stdout:2024-09-21T15:17:33.162+0000 7f7e253a8700 0 -- 172.21.15.146:0/4021574569 >> [v2:172.21.15.67:6808/1817390160,v1:172.21.15.67:6809/1817390160] conn(0x56529dd37c20 msgr2=0x56529dd3a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:17:35.182 INFO:tasks.workunit.client.0.smithi146.stdout:2024-09-21T15:17:33.282+0000 7f7e253a8700 0 -- 172.21.15.146:0/4021574569 >> [v2:172.21.15.67:6808/1817390160,v1:172.21.15.67:6809/1817390160] conn(0x7f7e0803af30 msgr2=0x7f7e0803d360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:17:35.182 INFO:tasks.workunit.client.0.smithi146.stdout:2024-09-21T15:17:34.578+0000 7f7e06ffd700 0 rbd-nbd: disconnect request received 2024-09-21T15:17:35.183 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.flUinkcHzA/rbd-nbd.log 2024-09-21T15:17:35.183 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep 'quiesce failed' /tmp/tmp.flUinkcHzA/rbd-nbd.log 2024-09-21T15:17:35.184 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:35.185 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd17425 2024-09-21T15:17:35.185 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:35.186 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:35.188 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:35.188 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:35.191 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:35.191 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:35.191 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd17425 2024-09-21T15:17:35.253 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:35.252+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:35.287 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:35.288 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:35.288 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:35.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:35.355 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:35.355+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:35.374 INFO:tasks.workunit.client.0.smithi146.stderr:+ OUT=/dev/nbd0 2024-09-21T15:17:35.374 INFO:tasks.workunit.client.0.smithi146.stderr:+ read DEV COOKIE 2024-09-21T15:17:35.375 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:35.375 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:35.375 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:35.375 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:35.375 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:35.379 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 2, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:35.379 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:35.379 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 3, 5, 2, 4] 2024-09-21T15:17:35.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-09-21T15:17:35.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.956081115000003, 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.908196857999997, 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.860025012000001, 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.501855910000003, 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.335872039000002, 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.150366769999998, 2024-09-21T15:17:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.171727939, 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.170943061999999, 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.170402938000002, 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.170200522000002, 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.170004317, 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:35.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.169810194, 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.482 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:35.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:35.487 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.115362973000003, 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:35.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:35.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:35.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.115230525999998, 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.115079147000003, 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:35.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:35.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:35.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.114951775999998, 2024-09-21T15:17:35.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:35.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:35.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:35.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.980151776, 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.510 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:35.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:35.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:35.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.843125692000001, 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:35.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:35.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:35.521 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.667828788000001, 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:35.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:35.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.617536262999998, 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:35.527 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:35.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:35.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.533 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:35.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19882 2024-09-21T15:17:35.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19882 2024-09-21T15:17:35.534 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19882 -C rbd-nbd 2024-09-21T15:17:35.535 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:35.535 INFO:tasks.workunit.client.0.smithi146.stdout: 19882 ? 00:00:00 rbd-nbd 2024-09-21T15:17:35.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:35.536 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:35.535+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:35.537 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:35.537 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:35.537 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:++ which mount 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/mount 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device detach rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:35.538 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:35.612 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false get_pid rbd 2024-09-21T15:17:35.612 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:35.613 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:35.613 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:35.613 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:35.613 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:35.635 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:35.632+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:35.699 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID= 2024-09-21T15:17:35.700 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n '' 2024-09-21T15:17:35.700 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 1 2024-09-21T15:17:35.700 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:35.700 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:35.700 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:35.700 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:35.700 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:35.700 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:35.701 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:35.701 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:35.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:35.702 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:35.704 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:35.704 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:35.704 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:35.705 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-09-21T15:17:35.703+0000 7f6b15ef9700 -1 mon.a@0(leader).osd e87 definitely_dead 0 2024-09-21T15:17:35.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:35.735 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:35.732+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:35.743 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: could not validate attach request 2024-09-21T15:17:35.743 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-21T15:17:35.743 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: must specify --cookie or --force to proceed 2024-09-21T15:17:35.746 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:35.746 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -n '' ']' 2024-09-21T15:17:35.746 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17425 --device-type nbd --force 2024-09-21T15:17:35.746 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:35.746 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:35.747 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:35.748 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:35.750 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:35.750 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:35.750 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd17425 --device-type nbd --force 2024-09-21T15:17:35.814 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:35.814 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:35.815 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:35.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:35.836 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:35.836+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:35.891 INFO:tasks.workunit.client.0.smithi146.stdout:/dev/nbd0 2024-09-21T15:17:35.901 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:35.901 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:35.901 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:35.902 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:35.902 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:35.937 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:35.935+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:35.939 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 138.157141083, 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:35.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:35.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:35.946 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.480203861, 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:35.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:35.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.152184492000004, 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:35.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:35.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:35.956 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.101023048999998, 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:35.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:35.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:35.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.012977083000003, 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.962 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:35.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:35.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.432445303999998, 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:35.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.026200621999998, 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:35.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.463416852000002, 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:35.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.380708079000001, 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.982 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:35.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:35.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:35.987 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.338588805000001, 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:35.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:35.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.192098303999998, 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.993 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:35.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:35.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:35.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.095792881999998, 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:35.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:35.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:36.001 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.54562241, 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:36.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:36.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:36.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.492783885999998, 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:36.007 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:36.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:36.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.012 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.446942374000002, 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:36.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.614852208999999, 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.018 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:36.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:36.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:36.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.611290647000001, 2024-09-21T15:17:36.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:36.026 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.607505473, 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:36.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:36.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:36.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.432470631000001, 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:36.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.544678251000001, 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:17:36.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:17:36.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:17:36.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:17:36.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:36.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:36.040 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:36.042 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=19953 2024-09-21T15:17:36.042 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 19953 2024-09-21T15:17:36.042 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 19953 -C rbd-nbd 2024-09-21T15:17:36.042 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:36.042 INFO:tasks.workunit.client.0.smithi146.stdout: 19953 ? 00:00:00 rbd-nbd 2024-09-21T15:17:36.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:36.044 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-21T15:17:36.044 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:36.040+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:36.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:36.096 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false get_pid rbd 2024-09-21T15:17:36.097 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:36.097 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:36.097 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:36.097 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:36.097 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:36.143 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:36.140+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:36.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID= 2024-09-21T15:17:36.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n '' 2024-09-21T15:17:36.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 1 2024-09-21T15:17:36.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:36.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -n '' ']' 2024-09-21T15:17:36.181 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd17425 --device-type nbd --force 2024-09-21T15:17:36.182 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:36.182 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:36.183 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:36.183 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:36.185 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:36.185 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:36.185 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd17425 --device-type nbd --force 2024-09-21T15:17:36.199 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:36.199 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:36.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:36.230 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:36.244 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:36.244+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:36.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:36.277 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:36.299 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:36.311 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:36.311 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:36.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:36.323 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:36.325 INFO:tasks.workunit.client.0.smithi146.stdout:/dev/nbd0 2024-09-21T15:17:36.337 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:36.338 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:36.338 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:36.338 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:36.338 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:36.346 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:36.346 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:36.347+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:36.348 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:36.348 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:36.349 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:36.349 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:36.349 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:36.349 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:17:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.3. 2024-09-21T15:17:36.424 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=20011 2024-09-21T15:17:36.426 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 20011 2024-09-21T15:17:36.427 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 20011 -C rbd-nbd 2024-09-21T15:17:36.434 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:36.435 INFO:tasks.workunit.client.0.smithi146.stdout: 20011 ? 00:00:00 rbd-nbd 2024-09-21T15:17:36.435 INFO:tasks.workunit.client.0.smithi146.stderr:+ ls /tmp/tmp.flUinkcHzA/mnt/ 2024-09-21T15:17:36.436 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:36.436 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:36.438 INFO:tasks.workunit.client.0.smithi146.stdout:lost+found 2024-09-21T15:17:36.438 INFO:tasks.workunit.client.0.smithi146.stdout:test 2024-09-21T15:17:36.438 INFO:tasks.workunit.client.0.smithi146.stderr:+ dd if=/tmp/tmp.flUinkcHzA/mnt/test of=/dev/null bs=1M count=1 2024-09-21T15:17:36.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:36.449 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:36.444+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:36.450 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:17:36.450 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-09-21T15:17:36.450 INFO:tasks.workunit.client.0.smithi146.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918684 s, 114 MB/s 2024-09-21T15:17:36.451 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo dd if=/tmp/tmp.flUinkcHzA/data of=/tmp/tmp.flUinkcHzA/mnt/test1 bs=1M count=1 oflag=direct 2024-09-21T15:17:36.451 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:36.451 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:36.452 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:36.452 INFO:tasks.workunit.client.0.smithi146.stderr:++ which dd 2024-09-21T15:17:36.454 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/dd 2024-09-21T15:17:36.454 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:36.454 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.flUinkcHzA/data of=/tmp/tmp.flUinkcHzA/mnt/test1 bs=1M count=1 oflag=direct 2024-09-21T15:17:36.528 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records in 2024-09-21T15:17:36.528 INFO:tasks.workunit.client.0.smithi146.stderr:1+0 records out 2024-09-21T15:17:36.528 INFO:tasks.workunit.client.0.smithi146.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656762 s, 16.0 MB/s 2024-09-21T15:17:36.529 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo umount /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:36.529 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:36.530 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:36.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:36.532 INFO:tasks.workunit.client.0.smithi146.stderr:++ which umount 2024-09-21T15:17:36.535 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/umount 2024-09-21T15:17:36.535 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:36.535 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:17:36.548 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:36.547+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:36.559 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd0 20011 2024-09-21T15:17:36.559 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd0 2024-09-21T15:17:36.559 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=20011 2024-09-21T15:17:36.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:17:36.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:36.560 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:36.561 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:36.562 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:36.564 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:36.564 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:36.564 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T15:17:36.571 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:36.571 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:36.581 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:36.648 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:36.647+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:36.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:17:36.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^20011\b' 2024-09-21T15:17:36.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^20011\b' 2024-09-21T15:17:36.689 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:36.689 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:36.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:36.746 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:36.747 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:17:36.747 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *20011\b' 2024-09-21T15:17:36.747 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *20011\b' 2024-09-21T15:17:36.750 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:36.747+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:36.756 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:36.757 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:17:36.803 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:36.804 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:36.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:36.850 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:36.851+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:36.922 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:36.922 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:36.932 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:36.951 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:36.948+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:37.041 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.041 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:37.053 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:37.048+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:37.153 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:37.151+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:37.158 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.158 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:37.254 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:37.255+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:37.258 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -n '' ']' 2024-09-21T15:17:37.258 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:37.259 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd snap ls rbd/testrbdnbd17425 2024-09-21T15:17:37.260 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-21T15:17:37.274 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.275 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.284 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:37.333 INFO:tasks.workunit.client.0.smithi146.stderr:+ SNAPID=4 2024-09-21T15:17:37.333 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd17425 2024-09-21T15:17:37.333 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:37.334 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:37.335 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:37.335 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:37.338 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:37.338 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:37.338 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd17425 2024-09-21T15:17:37.355 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:37.352+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:37.394 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.394 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:37.455 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:37.452+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:37.523 INFO:tasks.workunit.client.0.smithi146.stderr:+ OUT=/dev/nbd0 2024-09-21T15:17:37.523 INFO:tasks.workunit.client.0.smithi146.stderr:+ read DEV COOKIE 2024-09-21T15:17:37.523 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:37.524 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:37.524 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:37.525 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:37.525 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:37.556 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:37.556+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:37.560 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.560 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:37.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=20108 2024-09-21T15:17:37.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 20108 2024-09-21T15:17:37.605 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 20108 -C rbd-nbd 2024-09-21T15:17:37.611 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:37.611 INFO:tasks.workunit.client.0.smithi146.stdout: 20108 ? 00:00:00 rbd-nbd 2024-09-21T15:17:37.611 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device detach rbd/testrbdnbd17425 --snap-id 4 --device-type nbd 2024-09-21T15:17:37.612 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:37.612 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:37.613 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:37.613 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:37.615 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:37.616 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:37.616 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd17425 --snap-id 4 --device-type nbd 2024-09-21T15:17:37.657 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:37.656+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:37.677 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.677 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:37.718 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false get_pid rbd 2024-09-21T15:17:37.719 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:37.719 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:37.719 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:37.720 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:37.720 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:37.758 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:37.759+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:37.791 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID= 2024-09-21T15:17:37.791 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n '' 2024-09-21T15:17:37.791 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 1 2024-09-21T15:17:37.792 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:37.792 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:37.792 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:37.792 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:37.792 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:37.792 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.792 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.793 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:37.793 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:37.795 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17425 --device-type nbd 2024-09-21T15:17:37.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:37.837 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: could not validate attach request 2024-09-21T15:17:37.837 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-21T15:17:37.838 INFO:tasks.workunit.client.0.smithi146.stderr:rbd: must specify --cookie or --force to proceed 2024-09-21T15:17:37.840 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:37.840 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -n '' ']' 2024-09-21T15:17:37.840 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17425 --device-type nbd --force 2024-09-21T15:17:37.841 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:37.841 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:37.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:37.843 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:37.844 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:37.844 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:37.844 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd17425 --device-type nbd --force 2024-09-21T15:17:37.859 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:37.856+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:37.911 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:37.911 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:37.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:37.960 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:37.956+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:37.992 INFO:tasks.workunit.client.0.smithi146.stdout:/dev/nbd0 2024-09-21T15:17:38.004 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:38.004 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:38.004 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:38.005 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:38.005 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:38.038 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:38.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:38.061 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:38.060+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:38.095 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=20173 2024-09-21T15:17:38.095 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 20173 2024-09-21T15:17:38.096 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 20173 -C rbd-nbd 2024-09-21T15:17:38.108 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:17:38.108 INFO:tasks.workunit.client.0.smithi146.stdout: 20173 ? 00:00:00 rbd-nbd 2024-09-21T15:17:38.109 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-09-21T15:17:38.109 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:17:38.109 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:17:38.110 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:17:38.110 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:17:38.111 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:17:38.111 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:17:38.111 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-09-21T15:17:38.154 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:38.155 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:38.162 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:38.163+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:38.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:38.216 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false get_pid rbd 2024-09-21T15:17:38.216 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:17:38.216 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:17:38.216 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:17:38.216 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:17:38.217 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T15:17:38.263 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:38.263+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:38.295 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID= 2024-09-21T15:17:38.295 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n '' 2024-09-21T15:17:38.296 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 1 2024-09-21T15:17:38.296 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:17:38.296 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:17:38.296 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd config image set rbd/testrbdnbd17425 rbd_discard_granularity_bytes 4096 2024-09-21T15:17:38.314 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:38.314 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:38.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:38.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:38.363 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:38.363+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:38.404 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd feature enable rbd/testrbdnbd17425 journaling 2024-09-21T15:17:38.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:17:38.431 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:38.431 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:38.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:38.465 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:38.464+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:38.501 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:38.508 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:17:38.508 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:17:38.508 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:17:38.510 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:17:38.511 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:17:38.513 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:17:38.513 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:17:38.513 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:17:38.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:38.546 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:38.565 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:38.563+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:38.568 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:38.580 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:38.580 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:38.580 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:38.581 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:38.581 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:38.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:38.592 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:38.617 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:38.620 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:38.666 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:38.664+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:38.697 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:38.697 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:38.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:38.697 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 135.87416162100001, 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:38.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.311737119, 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:38.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:38.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.301000618000003, 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.707 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:38.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.256543784000002, 2024-09-21T15:17:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.715 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.248766981999999, 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:38.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:38.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.719 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.209593310999999, 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:38.723 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.202860950000002, 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:38.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.168809457999998, 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:38.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.158683474, 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.732 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.46192009, 2024-09-21T15:17:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:38.740 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.324540787, 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:38.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:38.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:38.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:38.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:38.745 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.323716735000001, 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:38.749 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.438655186999998, 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:38.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:38.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.753 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.437699116000001, 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:38.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:38.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.298605034000001, 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029330900000000001, 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397916+0000", 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397928+0000", 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1897e-05 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397948+0000", 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0319999999999999e-05 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397964+0000", 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5868000000000001e-05 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398041+0000", 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6278000000000004e-05 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398130+0000", 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9227999999999993e-05 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398212+0000", 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1964000000000003e-05 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.298023733000001, 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:38.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:38.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.00483199, 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000394242, 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.766 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691689+0000", 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691697+0000", 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.174e-06 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691708+0000", 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0492000000000001e-05 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691716+0000", 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5399999999999996e-06 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691813+0000", 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6644000000000005e-05 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691957+0000", 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014423300000000001 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692086+0000", 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012863100000000001 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:38.770 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.004110458, 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:38.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:38.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.364171718, 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.5557.0:362 2.0 e81/62 ondisk, result = 0)", 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.359430262, 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031135899999999998, 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337092+0000", 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337101+0000", 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6670000000000002e-06 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337120+0000", 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9623000000000002e-05 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337134+0000", 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3842000000000001e-05 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337214+0000", 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9992000000000003e-05 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337230+0000", 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6246999999999999e-05 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337405+0000", 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174193 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:38.784 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:38.786 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:38.787 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:38.783+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:38.887 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:38.884+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:38.902 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:38.903 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:38.903 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:38.903 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:38.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:38.988 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:38.988+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:39.017 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:39.018 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:39.018 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:39.018 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:39.018 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:39.027 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:39.089 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:39.087+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:39.140 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:39.140 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:39.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.513602609000003, 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:39.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:636 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'39, mlcod=57'37)", 2024-09-21T15:17:39.145 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.473156017000001, 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026757356999999999, 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665160+0000", 2024-09-21T15:17:39.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.665159+0000", 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666049+0000", 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089081700000000002 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666073+0000", 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3468999999999998e-05 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666083+0000", 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0462000000000001e-05 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666141+0000", 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.7946999999999999e-05 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.666166+0000", 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4739000000000001e-05 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691828+0000", 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025661732999999999 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.691917+0000", 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9306000000000002e-05 2024-09-21T15:17:39.149 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.397454672000002, 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:39.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:39.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.350834274999997, 2024-09-21T15:17:39.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:39.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:39.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.158 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.300217650999997, 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.159 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.211864179000003, 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:39.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.722733308999999, 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:39.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.007852466999999, 2024-09-21T15:17:39.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:39.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.175 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:708 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'55, mlcod=58'53)", 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.963145640999997, 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029089514, 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175170+0000", 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175168+0000", 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175918+0000", 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074915699999999997 2024-09-21T15:17:39.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175947+0000", 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9198999999999999e-05 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.175959+0000", 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1807000000000001e-05 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176032+0000", 2024-09-21T15:17:39.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.3635999999999998e-05 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.176049+0000", 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6422999999999998e-05 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204213+0000", 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028164112000000002 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.204260+0000", 2024-09-21T15:17:39.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6749999999999998e-05 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.919538209999999, 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:39.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:39.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:714 2.1 e58/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 58'5, mlcod=50'3)", 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.823229632999997, 2024-09-21T15:17:39.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028102708000000001, 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315086+0000", 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315084+0000", 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315669+0000", 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00058507399999999999 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315700+0000", 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0307e-05 2024-09-21T15:17:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315714+0000", 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4175999999999999e-05 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315781+0000", 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6755999999999996e-05 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.315817+0000", 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6220000000000002e-05 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343145+0000", 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027327939999999998 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343189+0000", 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4002000000000003e-05 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:723 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'15, mlcod=57'13)", 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.662120234, 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028631713999999999, 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476195+0000", 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.476194+0000", 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477123+0000", 2024-09-21T15:17:39.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092921200000000001 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477151+0000", 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7483000000000001e-05 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477162+0000", 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1909999999999999e-05 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477219+0000", 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6224000000000002e-05 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.477243+0000", 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3788e-05 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504732+0000", 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027489164 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.504827+0000", 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.5456000000000002e-05 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:725 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'17, mlcod=59'15)", 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.616789764000004, 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027755291000000001, 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521526+0000", 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.521524+0000", 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522339+0000", 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000814169 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522367+0000", 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8347000000000002e-05 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522381+0000", 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.401e-05 2024-09-21T15:17:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522466+0000", 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4777000000000005e-05 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.522501+0000", 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5073000000000001e-05 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549170+0000", 2024-09-21T15:17:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026669305000000001 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.549281+0000", 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011113299999999999 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.537566380999998, 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.201 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:44 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'94, mlcod=61'92)", 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.704093387, 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028380540999999999, 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434222+0000", 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.434220+0000", 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435653+0000", 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0014329519999999999 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435684+0000", 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0703000000000001e-05 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435702+0000", 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7730000000000001e-05 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435760+0000", 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8263000000000001e-05 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.435789+0000", 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462568+0000", 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026779311 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.462603+0000", 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4912e-05 2024-09-21T15:17:39.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.651927932, 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:39.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.210 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:48 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'98, mlcod=62'96)", 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.607137021, 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027676580999999999, 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531179+0000", 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.531177+0000", 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532021+0000", 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00084430300000000002 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532050+0000", 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8873e-05 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532067+0000", 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7407000000000001e-05 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532161+0000", 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3245000000000006e-05 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.532174+0000", 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3699e-05 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558812+0000", 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026637392999999999 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.558855+0000", 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3520999999999999e-05 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.214 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:50 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'100, mlcod=62'98)", 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.567613952999999, 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028652052000000001, 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570702+0000", 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.570700+0000", 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571610+0000", 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00091016899999999995 2024-09-21T15:17:39.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571637+0000", 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6777000000000001e-05 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571653+0000", 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.63e-05 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571708+0000", 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4923000000000003e-05 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.571736+0000", 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8259000000000002e-05 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599298+0000", 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027561602000000001 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.599354+0000", 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5884999999999998e-05 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:165 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'18, mlcod=63'16)", 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.834476375000001, 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026625488999999999, 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.219 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303839+0000", 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303838+0000", 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304203+0000", 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00036489299999999998 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304228+0000", 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5511999999999999e-05 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304245+0000", 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6991e-05 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304304+0000", 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8941999999999999e-05 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.304323+0000", 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8802999999999999e-05 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330424+0000", 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026101201000000001 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330465+0000", 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0643999999999999e-05 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:39.223 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.800192602999999, 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.227 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:39.229 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:39.229 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:39.228+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:39.330 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:39.331+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:39.343 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:39.343 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:39.343 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:39.353 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:39.430 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:39.431+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:17:39.462 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:39.462 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:39.462 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:39.462 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:39.462 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:39.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:39.532 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:39.528+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.317150787000003, 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:39.583 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.242925571000001, 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.232526131999997, 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.170219125999999, 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 36.108009058, 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.602 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.410873821999999, 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.603 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:39.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:39.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:39.608 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.410423844, 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:39.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:39.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:39.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.410173979, 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.614 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:39.619 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.409898474999999, 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:39.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.624 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.409495509999999, 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.625 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:39.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:39.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:39.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.409289474000001, 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:39.630 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:39.635 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.409085471000001, 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:39.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:39.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:39.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:39.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.408894588999999, 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:39.641 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.646 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.271831262999999, 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.647 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:17:39.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:17:39.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:17:39.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.271596241000001, 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.652 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.271395817999998, 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:17:39.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:17:39.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:17:39.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.270992236000001, 2024-09-21T15:17:39.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.270733078999999, 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:17:39.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:17:39.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:17:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.270537852, 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:17:39.674 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:17:39.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.076286088, 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:39.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:39.685 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:39.686 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:39.687 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:39.687+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:39.787 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:39.784+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:39.838 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:39.839 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:39.839 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:39.839 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:39.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:39.888 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:39.887+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:39.958 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:39.958 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:39.958 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:39.958 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:39.958 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:39.968 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:39.988 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:39.988+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:40.077 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:40.077 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:40.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:40.077 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:40.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.077 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.972037202999999, 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.078 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:40.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:40.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:40.081 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.97198616, 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.082 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.956366617, 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:40.090 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.956339864, 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.091 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:40.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.787291760999999, 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.095 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.096 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:40.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:40.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:40.101 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.772146904000003, 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.102 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:40.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:40.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:40.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.743000088000002, 2024-09-21T15:17:40.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:40.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:40.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.733673183000001, 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.113 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:40.118 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.706037031000001, 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.119 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:40.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.124 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.694959326000003, 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:40.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.660189779, 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:40.130 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:40.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.771665697000003, 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.136 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:40.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:40.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.76141063, 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.142 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:40.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:40.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.729363632999998, 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:40.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.153 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.717151680000001, 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:40.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:40.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.683255721999998, 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.159 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:40.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:40.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:40.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.644210852999997, 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.165 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:40.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.170 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.783767726000001, 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.171 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:40.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:40.177 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.772190219000002, 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:40.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 42.737779451000002, 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:40.183 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:40.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:40.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:40.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:40.189 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:40.189 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:40.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:40.190 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:40.191+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:40.291 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:40.288+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:40.310 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:40.310 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:40.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:40.392 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:40.392+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:40.434 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:40.434 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:40.435 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:40.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:40.493 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:40.491+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:40.559 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:40.559 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.099857577000002, 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:40.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.051973320000002, 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:40.565 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:40.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.003801473999999, 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:40.571 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:40.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:40.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.645632372000001, 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:40.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.479648501, 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:40.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:40.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.294143232000003, 2024-09-21T15:17:40.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:40.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:40.593 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.315504400999998, 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.594 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.599 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.314719523999997, 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.3141794, 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:40.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:40.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.313976984, 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:40.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.313780778999998, 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.617 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:40.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:40.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:40.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.313586655999998, 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:40.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:40.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:40.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.259139435000002, 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:40.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:40.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.259006988000003, 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:40.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:40.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.258855609000001, 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:40.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:40.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.258728238000003, 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:40.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:40.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:40.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.650 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.123928237999998, 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:40.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.986902153999999, 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:40.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:40.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:40.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.81160525, 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:40.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:40.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.761312725000003, 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:40.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:40.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:40.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:40.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:40.673 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:40.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:40.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:40.675 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:40.675+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:40.775 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:40.775+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:40.833 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:40.834 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:40.834 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:40.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:40.862 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:40.876 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:40.872+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:40.884 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:40.903 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:40.925 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:40.947 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:40.947 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:40.947 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:40.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:40.971 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:40.972 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:40.972 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:40.976 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:40.975+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:41.064 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.064 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:41.077 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:41.075+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:41.177 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:41.176+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:41.180 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.180 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:41.278 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:41.279+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:41.296 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.296 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:41.379 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:41.376+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:41.430 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:41.430 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:41.431 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-21T15:17:41.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-09-21T15:17:41.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:17:41.466 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.466 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:41.522 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:41.523+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:41.581 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:41.579+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:41.594 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.594 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.604 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:41.682 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:41.683+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:41.710 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.710 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.720 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:41.736 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-09-21T15:17:41.735+0000 7f6b15ef9700 -1 mon.a@0(leader).osd e90 definitely_dead 0 2024-09-21T15:17:41.783 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:41.780+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:41.827 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.827 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.836 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:41.884 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:41.880+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:41.984 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:41.983+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:41.989 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:41.989 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:41.999 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:42.085 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:42.083+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:42.104 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.104 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.114 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:42.186 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:42.184+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:42.222 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.222 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.231 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:42.287 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:42.287+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:42.380 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.380 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.387 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:42.384+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:42.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.3. 2024-09-21T15:17:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:17:42.487 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-21T15:17:42.494 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.495 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.504 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:42.530 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:42.531+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:42.589 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:42.588+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:42.608 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.608 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:42.689 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:42.688+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:42.735 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.736 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:42.790 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:42.791+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:42.852 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.852 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:42.890 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:42.891+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:42.967 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:42.967 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:42.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:42.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:42.991 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:42.988+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:43.092 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:43.091+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:43.126 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:43.126 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:43.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:43.160 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:43.193 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:43.191+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:43.197 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:43.232 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:43.269 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:43.272 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:43.272 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:43.272 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:43.272 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:43.272 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:43.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:43.291 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:43.314 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:43.314 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:43.315+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:43.316 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:43.316 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:43.316 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:43.316 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:43.316 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:43.316 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:43.395 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:43.395+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:43.397 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:43.397 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:43.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:43.397 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:43.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 145.61586495399999, 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.938927732000003, 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.610908363, 2024-09-21T15:17:43.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:43.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.412 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:43.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:43.412 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:43.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.413 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:43.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:43.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:43.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:43.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:43.418 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.559746920000002, 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:43.419 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:43.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.471700953999999, 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.425 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:43.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:43.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.891169175000002, 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:43.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.484924493000001, 2024-09-21T15:17:43.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:43.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.439 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.922140722999998, 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.440 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:43.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:43.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.839431949999998, 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.446 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:43.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:43.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.797312675999997, 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:43.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.650822175000002, 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.458 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:43.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:43.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.554516753000001, 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:43.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.004346280999997, 2024-09-21T15:17:43.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:43.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:43.473 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.951507757000002, 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:43.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.905666244999999, 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:43.479 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:43.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:43.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:43.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.073576079999999, 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:43.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:43.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.070014518000001, 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:43.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:43.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.066229344, 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:43.499 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.891194502000001, 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:43.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.003402122000001, 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:43.504 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:17:43.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:17:43.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:17:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.510 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.510 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:43.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:43.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:43.511+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:43.612 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:43.611+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:43.629 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:43.629 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:43.629 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:43.629 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:43.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:43.713 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:43.711+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:43.792 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:43.793 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:43.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:43.814 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:43.812+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:43.911 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 141.08982869799999, 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.527404195999999, 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.516667695000002, 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:43.924 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.472210861000001, 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:43.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.464434058999998, 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:43.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:43.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.425260387999998, 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.933 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:43.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:43.937 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.418528027000001, 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:43.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:43.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.384476534999997, 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:43.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:43.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.374350550999999, 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.677587166999999, 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.950 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:43.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.540207864000003, 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:43.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.539383811999997, 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:43.959 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.654322264000001, 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.963 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:43.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:43.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:43.967 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.653366192999997, 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:43.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.971 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.514272110999997, 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029330900000000001, 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397918+0000", 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397916+0000", 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397928+0000", 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1897e-05 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397948+0000", 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0319999999999999e-05 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.397964+0000", 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5868000000000001e-05 2024-09-21T15:17:43.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398041+0000", 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6278000000000004e-05 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398130+0000", 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9227999999999993e-05 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398212+0000", 2024-09-21T15:17:43.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1964000000000003e-05 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.51369081, 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:43.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:43.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:43.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.220499066999999, 2024-09-21T15:17:43.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000394242, 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691692+0000", 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691689+0000", 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691697+0000", 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.174e-06 2024-09-21T15:17:43.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691708+0000", 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0492000000000001e-05 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691716+0000", 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5399999999999996e-06 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691813+0000", 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6644000000000005e-05 2024-09-21T15:17:43.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.691957+0000", 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014423300000000001 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692086+0000", 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012863100000000001 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.219777534999999, 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:43.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:43.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.579838795000001, 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:43.989 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:17:43.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop_reply(client.5557.0:362 2.0 e81/62 ondisk, result = 0)", 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.575097338999999, 2024-09-21T15:17:43.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031135899999999998, 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337093+0000", 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337092+0000", 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337101+0000", 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6670000000000002e-06 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337120+0000", 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9623000000000002e-05 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337134+0000", 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3842000000000001e-05 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337214+0000", 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9992000000000003e-05 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337230+0000", 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6246999999999999e-05 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:43.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337405+0000", 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174193 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:43.999 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:44.001 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:44.001 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:43.999+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:44.101 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:44.103+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:44.125 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:44.125 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:44.125 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:44.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:44.202 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:44.203+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:44.231 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:44.231 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:44.231 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:44.231 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:44.231 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:44.241 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:44.302 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:44.303+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:44.348 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:44.348 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.721773312000003, 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:44.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:44.353 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.605625375000002, 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:44.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:44.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.559004977999997, 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:44.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:44.361 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.508388353999997, 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:44.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.420034882000003, 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:44.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:44.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.930904011999999, 2024-09-21T15:17:44.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:44.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:44.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.21602317, 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:44.379 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.127708912999999, 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 58.745737083999998, 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.384 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:44.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:44.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 49.860098635, 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:44.388 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:44.392 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.008363306, 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.85857144600000002, 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029737039, 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487914+0000", 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489004+0000", 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010899900000000001 2024-09-21T15:17:44.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489031+0000", 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7645e-05 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489053+0000", 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489151+0000", 2024-09-21T15:17:44.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7928000000000002e-05 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489165+0000", 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3451999999999999e-05 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.517652+0000", 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028487375999999998 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.79622133900000003, 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:17:44.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.53401746699999997, 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:44.408 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.50096750899999998, 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:44.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:44.412 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.41211023299999999, 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:44.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.414 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.415 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:44.416 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.34285676700000001, 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.417 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:17:44.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:17:44.419 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.29896919900000002, 2024-09-21T15:17:44.420 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.421 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:17:44.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.25408849, 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:17:44.424 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.425 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.426 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.16165791500000001, 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:17:44.428 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.429 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.430 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.432 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:44.433 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:44.434 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:44.431+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:44.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:17:44.534 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:44.535+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:44.544 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:44.544 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:44.544 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:44.544 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:44.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:44.635 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:44.632+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:44.658 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:44.659 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:44.659 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:44.659 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:44.659 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:44.668 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:44.736 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:44.735+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:44.827 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.561586126000002, 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:44.828 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:44.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:44.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:44.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:44.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:44.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:44.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:44.834 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.834 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.834 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.835 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:44.835 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:44.835 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.48736091, 2024-09-21T15:17:44.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:44.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.476961471000003, 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:44.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:44.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:44.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.844 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.414654464999998, 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:44.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.352444396999999, 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:44.848 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:44.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:44.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:44.852 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.655309161000002, 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:44.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:44.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.654859182999999, 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:44.858 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:44.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.864 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.654609317999999, 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:44.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:44.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.654333814000001, 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:44.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:44.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:44.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.653930848999998, 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.653724813, 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.65352081, 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:44.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:44.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:44.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.893 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.653329928000002, 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:44.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:44.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.899 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.516266602000002, 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.904 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.51603158, 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.905 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:17:44.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.515831157000001, 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.910 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:17:44.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:17:44.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:44.915 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.515427575, 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:17:44.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:44.921 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.515168418000002, 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.514973190999999, 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.928 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:44.933 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 27.320721426999999, 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:44.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:44.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:17:44.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:17:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.939 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.939 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.939 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:44.939 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:44.939 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:44.940 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:44.941 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:44.940+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:45.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:45.039+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:45.046 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:45.046 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:45.046 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:45.046 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:45.056 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:45.142 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:45.143+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:45.165 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:45.165 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:45.165 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:45.165 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:45.165 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:45.175 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:45.243 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:45.240+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:45.293 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:45.293 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:45.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.188231779999995, 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:45.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:45.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:45.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:45.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:45.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:45.298 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.299 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.299 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.299 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:45.299 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:45.299 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.188180736999996, 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:45.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.172561193999996, 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:45.304 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:45.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.172534440999996, 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:45.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 61.003486338000002, 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.313 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:45.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.988341480999999, 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:45.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.959194664999998, 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.325 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:45.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:45.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:45.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.949867759999997, 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:45.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.922231607999997, 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:45.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:45.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:45.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.349 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.911153902999999, 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:45.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.876384356000003, 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.356 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:45.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.987860273999999, 2024-09-21T15:17:45.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:45.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.977605207000003, 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.369 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:45.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:45.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.945558210000002, 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:45.374 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:45.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:45.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.379 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.933346256999997, 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:45.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.899450299000002, 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.385 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.390 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.86040543, 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:45.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:45.396 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.999962302999997, 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:45.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:45.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:45.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.988384795999998, 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.403 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:45.405 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:45.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:45.408 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 47.953974027999998, 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.410 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.412 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:45.414 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:45.414 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:45.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:45.416 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:45.412+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:45.416 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:17:45.516 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:45.515+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:45.539 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:45.540 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:45.540 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:45.540 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:45.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:45.568 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:45.587 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:45.606 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:45.617 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:45.616+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:45.630 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:45.652 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:45.652 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:45.653 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:45.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:45.677 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:45.679 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:45.679 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:45.679 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:45.679 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:45.679 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:45.730 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:45.731+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:45.782 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:45.783 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:45.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:45.818 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:45.816+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:45.899 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:45.899 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:45.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:45.954 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:45.955+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:46.020 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:46.019+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:46.020 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.020 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:46.121 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:46.120+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:46.133 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.134 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:46.222 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:46.223+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:46.250 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.251 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:46.323 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:46.323+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:46.414 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.414 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.423 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:46.420+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:46.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:46.524 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:46.523+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:46.530 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.531 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:46.626 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:46.624+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:46.658 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.658 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.668 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:46.727 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:46.727+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:46.777 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.777 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.787 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:46.828 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:46.827+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:46.891 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:46.892 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:46.901 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:46.929 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:46.928+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:47.009 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:47.010 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:47.019 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:47.029 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:47.027+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:47.123 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:47.123 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:47.130 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:47.131+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:47.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:47.231 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:47.231+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:47.236 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:47.236 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:47.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:47.332 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:47.332+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:47.354 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:47.354 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:47.363 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:47.433 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:47.431+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:47.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:47.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:47.488 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T15:17:47.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:17:47.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:17:47.514 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:47.514 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:47.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:47.533 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:47.532+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:47.635 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:47.632+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:47.635 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:47.635 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:47.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:47.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:47.736 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:47.736+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:47.777 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:47.777 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:47.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:47.837 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:47.835+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 93, 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:17:45.470494+0000", 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:17:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:17:42.447038+0000", 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5801984, 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:17:47.848 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:17:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:17:47.850 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:47.851 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:47.852 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:47.853 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:47.853 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:47.853 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:47.853 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:47.853 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:47.854 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:17:47.855 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:47.856 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:47.857 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "87", 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:47.858 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 8, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 87, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:47.859 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:47.860 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.25, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.25, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2.25, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2.25, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:47.861 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "new_default_pool17425", 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:16:34.640599+0000", 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:47.862 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:47.863 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "54", 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 54, 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:47.864 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.2699999809265137, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.2699999809265137, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 68, 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.870 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.871 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:47.873 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 80, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.874 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.875 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:17:47.876 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:17:47.877 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:47.878 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 91, 2024-09-21T15:17:47.879 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.880 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.883 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:47.884 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:17:47.885 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:17:47.886 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 91, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.891 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:17:47.896 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:17:47.897 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:17:47.898 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:47.899 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:47.900 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:47.901 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:47.902 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:47.904 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.1", 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: 4, 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: 5 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.1", 2024-09-21T15:17:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: 2, 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: 3 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.4", 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 4 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.906 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:47.907 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:47.908 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:47.910 INFO:tasks.thrashosds.thrasher:j is {'epoch': 93, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:17:45.470494+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:17:42.447038+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'new_default_pool17425', 'create_time': '2024-09-21T15:16:34.640599+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 68, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.1', 'osds': [4, 5]}, {'pgid': '3.1', 'osds': [2, 3]}], 'pg_upmap_items': [{'pgid': '2.4', 'mappings': [{'from': 5, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:17:47.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:17:47.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:47.911 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:47.911 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:47.911 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:47.911 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:47.911 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:47.911 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:47.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:47.938 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:47.939+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:47.948 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:47.971 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:47.995 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:48.025 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:48.025 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:48.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:48.025 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:48.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.025 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:48.025 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.567175078999995, 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:48.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:48.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.519290822000002, 2024-09-21T15:17:48.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:48.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.471118976, 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.037 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:48.042 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.112949874000002, 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:48.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:48.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:48.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.946966003, 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.048 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:48.053 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.761460734000003, 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:48.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:48.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:48.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.782821902999999, 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:48.064 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.782037025999998, 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:48.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:48.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:48.069 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.781496902000001, 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:48.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:48.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:48.075 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.781294486, 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:48.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:48.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:48.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:48.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.781098280999998, 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.081 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:48.086 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.780904157999998, 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:48.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:48.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:48.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:48.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.726456937000002, 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:48.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.726324490000003, 2024-09-21T15:17:48.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.102 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.726173111000001, 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.103 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:48.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:48.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.726045740000004, 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:48.108 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:48.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.591245739999998, 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.454219655999999, 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:48.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:48.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:48.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:48.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.278922752, 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:48.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.228630226999996, 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.130 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:48.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:48.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:48.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.136 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:48.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:48.137 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:48.135+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:48.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:48.200 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:17:48.213 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-21T15:17:47.956341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147465123,"num_objects":77,"num_object_clones":19,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":13,"num_read":5722,"num_read_kb":155475,"num_write":936,"num_write_kb":273253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":155189515,"num_keys_recovered":45,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":890,"ondisk_log_size":890,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":35,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":463912,"kb_used_data":277200,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":565767128,"statfs":{"total":579820584960,"available":579345539072,"internally_reserved":0,"allocated":283852800,"data_stored":301829492,"data_compressed":25932,"data_compressed_allocated":368640,"data_compressed_original":9846784,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":25165894,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001463"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":238,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.469180+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:17:43.469180+0000","last_peered":"2024-09-21T15:17:43.469180+0000","last_clean":"2024-09-21T15:17:43.469180+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:17:43.469180+0000","last_undegraded":"2024-09-21T15:17:43.469180+0000","last_fullsized":"2024-09-21T15:17:43.469180+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'45","reported_seq":1134,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.469241+0000","last_change":"2024-09-21T15:16:55.672969+0000","last_active":"2024-09-21T15:17:43.469241+0000","last_peered":"2024-09-21T15:17:43.469241+0000","last_clean":"2024-09-21T15:17:43.469241+0000","last_became_active":"2024-09-21T15:16:55.272135+0000","last_became_peered":"2024-09-21T15:16:55.272135+0000","last_unstale":"2024-09-21T15:17:43.469241+0000","last_undegraded":"2024-09-21T15:17:43.469241+0000","last_fullsized":"2024-09-21T15:17:43.469241+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:58.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":20971558,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":960,"num_read_kb":31831,"num_write":42,"num_write_kb":32833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"75'6","reported_seq":91,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:44.260941+0000","last_change":"2024-09-21T15:17:44.260941+0000","last_active":"2024-09-21T15:17:44.260941+0000","last_peered":"2024-09-21T15:17:44.260941+0000","last_clean":"2024-09-21T15:17:44.260941+0000","last_became_active":"2024-09-21T15:17:43.480798+0000","last_became_peered":"2024-09-21T15:17:43.480798+0000","last_unstale":"2024-09-21T15:17:44.260941+0000","last_undegraded":"2024-09-21T15:17:44.260941+0000","last_fullsized":"2024-09-21T15:17:44.260941+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:07.595836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181755,"stat_sum":{"num_bytes":54,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":127,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"82'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-21T15:17:42.466677+0000","last_change":"2024-09-21T15:16:55.569580+0000","last_active":"2024-09-21T15:17:42.466677+0000","last_peered":"2024-09-21T15:17:42.466677+0000","last_clean":"2024-09-21T15:17:42.466677+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:17:42.466677+0000","last_undegraded":"2024-09-21T15:17:42.466677+0000","last_fullsized":"2024-09-21T15:17:42.466677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.481624+0000","last_change":"2024-09-21T15:17:43.481624+0000","last_active":"2024-09-21T15:17:43.481624+0000","last_peered":"2024-09-21T15:17:43.481624+0000","last_clean":"2024-09-21T15:17:43.481624+0000","last_became_active":"2024-09-21T15:17:43.481181+0000","last_became_peered":"2024-09-21T15:17:43.481181+0000","last_unstale":"2024-09-21T15:17:43.481624+0000","last_undegraded":"2024-09-21T15:17:43.481624+0000","last_fullsized":"2024-09-21T15:17:43.481624+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:03:25.583980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"85'22","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-21T15:17:42.460830+0000","last_change":"2024-09-21T15:17:01.739071+0000","last_active":"2024-09-21T15:17:42.460830+0000","last_peered":"2024-09-21T15:17:42.460830+0000","last_clean":"2024-09-21T15:17:42.460830+0000","last_became_active":"2024-09-21T15:17:01.316522+0000","last_became_peered":"2024-09-21T15:17:01.316522+0000","last_unstale":"2024-09-21T15:17:42.460830+0000","last_undegraded":"2024-09-21T15:17:42.460830+0000","last_fullsized":"2024-09-21T15:17:42.460830+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:15.523790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":35,"num_read_kb":16384,"num_write":21,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.466036+0000","last_change":"2024-09-21T15:16:55.272512+0000","last_active":"2024-09-21T15:17:43.466036+0000","last_peered":"2024-09-21T15:17:43.466036+0000","last_clean":"2024-09-21T15:17:43.466036+0000","last_became_active":"2024-09-21T15:16:55.271874+0000","last_became_peered":"2024-09-21T15:16:55.271874+0000","last_unstale":"2024-09-21T15:17:43.466036+0000","last_undegraded":"2024-09-21T15:17:43.466036+0000","last_fullsized":"2024-09-21T15:17:43.466036+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:54:48.789474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046520300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"87'25","reported_seq":278,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.476633+0000","last_change":"2024-09-21T15:17:01.768225+0000","last_active":"2024-09-21T15:17:43.476633+0000","last_peered":"2024-09-21T15:17:43.476633+0000","last_clean":"2024-09-21T15:17:43.476633+0000","last_became_active":"2024-09-21T15:17:01.327838+0000","last_became_peered":"2024-09-21T15:17:01.327838+0000","last_unstale":"2024-09-21T15:17:43.476633+0000","last_undegraded":"2024-09-21T15:17:43.476633+0000","last_fullsized":"2024-09-21T15:17:43.476633+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:08.047427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":12582969,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":132,"num_read_kb":8306,"num_write":24,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'1","reported_seq":67,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.476024+0000","last_change":"2024-09-21T15:16:55.676403+0000","last_active":"2024-09-21T15:17:43.476024+0000","last_peered":"2024-09-21T15:17:43.476024+0000","last_clean":"2024-09-21T15:17:43.476024+0000","last_became_active":"2024-09-21T15:16:55.274451+0000","last_became_peered":"2024-09-21T15:16:55.274451+0000","last_unstale":"2024-09-21T15:17:43.476024+0000","last_undegraded":"2024-09-21T15:17:43.476024+0000","last_fullsized":"2024-09-21T15:17:43.476024+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:20.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046455,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"75'1","reported_seq":65,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:44.264959+0000","last_change":"2024-09-21T15:17:44.264959+0000","last_active":"2024-09-21T15:17:44.264959+0000","last_peered":"2024-09-21T15:17:44.264959+0000","last_clean":"2024-09-21T15:17:44.264959+0000","last_became_active":"2024-09-21T15:17:43.481232+0000","last_became_peered":"2024-09-21T15:17:43.481232+0000","last_unstale":"2024-09-21T15:17:44.264959+0000","last_undegraded":"2024-09-21T15:17:44.264959+0000","last_fullsized":"2024-09-21T15:17:44.264959+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:58.858880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"85'27","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:44.249295+0000","last_change":"2024-09-21T15:17:44.249295+0000","last_active":"2024-09-21T15:17:44.249295+0000","last_peered":"2024-09-21T15:17:44.249295+0000","last_clean":"2024-09-21T15:17:44.249295+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:17:44.249295+0000","last_undegraded":"2024-09-21T15:17:44.249295+0000","last_fullsized":"2024-09-21T15:17:44.249295+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":52,"num_read_kb":8203,"num_write":26,"num_write_kb":32770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"75'1","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-21T15:17:42.461995+0000","last_change":"2024-09-21T15:17:01.318184+0000","last_active":"2024-09-21T15:17:42.461995+0000","last_peered":"2024-09-21T15:17:42.461995+0000","last_clean":"2024-09-21T15:17:42.461995+0000","last_became_active":"2024-09-21T15:17:01.317913+0000","last_became_peered":"2024-09-21T15:17:01.317913+0000","last_unstale":"2024-09-21T15:17:42.461995+0000","last_undegraded":"2024-09-21T15:17:42.461995+0000","last_fullsized":"2024-09-21T15:17:42.461995+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:47:53.254684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043867299999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"89'174","reported_seq":1329,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.469224+0000","last_change":"2024-09-21T15:17:24.302020+0000","last_active":"2024-09-21T15:17:43.469224+0000","last_peered":"2024-09-21T15:17:43.469224+0000","last_clean":"2024-09-21T15:17:43.469224+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:17:43.469224+0000","last_undegraded":"2024-09-21T15:17:43.469224+0000","last_fullsized":"2024-09-21T15:17:43.469224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":27287606,"num_objects":14,"num_object_clones":6,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":1248,"num_read_kb":38813,"num_write":192,"num_write_kb":52364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.480003+0000","last_change":"2024-09-21T15:17:43.480003+0000","last_active":"2024-09-21T15:17:43.480003+0000","last_peered":"2024-09-21T15:17:43.480003+0000","last_clean":"2024-09-21T15:17:43.480003+0000","last_became_active":"2024-09-21T15:17:43.479431+0000","last_became_peered":"2024-09-21T15:17:43.479431+0000","last_unstale":"2024-09-21T15:17:43.480003+0000","last_undegraded":"2024-09-21T15:17:43.480003+0000","last_fullsized":"2024-09-21T15:17:43.480003+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:19.528788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092366200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"89'244","reported_seq":1245,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.471208+0000","last_change":"2024-09-21T15:17:05.211552+0000","last_active":"2024-09-21T15:17:43.471208+0000","last_peered":"2024-09-21T15:17:43.471208+0000","last_clean":"2024-09-21T15:17:43.471208+0000","last_became_active":"2024-09-21T15:17:05.209676+0000","last_became_peered":"2024-09-21T15:17:05.209676+0000","last_unstale":"2024-09-21T15:17:43.471208+0000","last_undegraded":"2024-09-21T15:17:43.471208+0000","last_fullsized":"2024-09-21T15:17:43.471208+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:02:52.206876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18923558,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":2396,"num_read_kb":34834,"num_write":141,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":20,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.476863+0000","last_change":"2024-09-21T15:17:43.476863+0000","last_active":"2024-09-21T15:17:43.476863+0000","last_peered":"2024-09-21T15:17:43.476863+0000","last_clean":"2024-09-21T15:17:43.476863+0000","last_became_active":"2024-09-21T15:17:43.476485+0000","last_became_peered":"2024-09-21T15:17:43.476485+0000","last_unstale":"2024-09-21T15:17:43.476863+0000","last_undegraded":"2024-09-21T15:17:43.476863+0000","last_fullsized":"2024-09-21T15:17:43.476863+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:37.900423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"89'222","reported_seq":477,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-21T15:17:43.469536+0000","last_change":"2024-09-21T15:17:24.303933+0000","last_active":"2024-09-21T15:17:43.469536+0000","last_peered":"2024-09-21T15:17:43.469536+0000","last_clean":"2024-09-21T15:17:43.469536+0000","last_became_active":"2024-09-21T15:17:24.303534+0000","last_became_peered":"2024-09-21T15:17:24.303534+0000","last_unstale":"2024-09-21T15:17:43.469536+0000","last_undegraded":"2024-09-21T15:17:43.469536+0000","last_fullsized":"2024-09-21T15:17:43.469536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:39.035665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":20971596,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":4,"num_read":719,"num_read_kb":16966,"num_write":321,"num_write_kb":53462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":159,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146874685,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":13,"num_read":5568,"num_read_kb":155345,"num_write":771,"num_write_kb":270831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":155189356,"num_keys_recovered":29,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277250048,"data_stored":298156666,"data_compressed":14242,"data_compressed_allocated":294912,"data_compressed_original":8699904,"omap_allocated":0,"internal_metadata":0},"log_size":764,"ondisk_log_size":764,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52712,"kb_used_data":21592,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94319128,"statfs":{"total":96636764160,"available":96582787072,"internally_reserved":0,"allocated":22110208,"data_stored":21975591,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.67100000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.133,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.85799999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.032,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.82699999999999996,"15min":0.83699999999999997},"min":{"1min":0.53100000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2070000000000001,"5min":1.212,"15min":1.212},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65300000000000002,"15min":0.64500000000000002},"min":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84699999999999998,"15min":0.84999999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0880000000000001,"5min":1.208,"15min":1.208},"last":0.80300000000000005}]},{"osd":4,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.82399999999999995,"15min":0.83599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0089999999999999,"5min":1.629,"15min":1.629},"last":0.754},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.77300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56764,"kb_used_data":25648,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315076,"statfs":{"total":96636764160,"available":96578637824,"internally_reserved":0,"allocated":26263552,"data_stored":25584571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.99099999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.83099999999999996},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96199999999999997,"5min":1.333,"15min":1.333},"last":1.014}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.115,"5min":1.133,"15min":1.133},"last":0.53800000000000003}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.64200000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.84299999999999997,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.038,"5min":1.581,"15min":1.581},"last":0.67400000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.0940000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81299999999999994,"15min":0.81499999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.86599999999999999,"15min":0.86499999999999999},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.1659999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92792,"kb_used_data":61672,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94279048,"statfs":{"total":96636764160,"available":96541745152,"internally_reserved":0,"allocated":63152128,"data_stored":73492029,"data_compressed":12966,"data_compressed_allocated":184320,"data_compressed_original":4923392,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.42899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.58799999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.90300000000000002,"5min":1.226,"15min":1.226},"last":0.751},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.73699999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84699999999999998,"15min":0.84599999999999997},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.272,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86499999999999999,"15min":0.85999999999999999},"min":{"1min":0.64900000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.80500000000000005}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":1.2589999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82408,"kb_used_data":51292,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94289432,"statfs":{"total":96636764160,"available":96552378368,"internally_reserved":0,"allocated":52523008,"data_stored":51843196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57799999999999996,"15min":0.59199999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.88300000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95499999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.66200000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91300000000000003,"5min":1.101,"15min":1.101},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.90100000000000002,"15min":0.91500000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.085,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.70199999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.90400000000000003,"15min":0.92200000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0289999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.91900000000000004,"15min":0.92900000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1839999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.82999999999999996}]}]},{"osd":4,"up_from":12,"seq":51539607587,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122464,"kb_used_data":91344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94249376,"statfs":{"total":96636764160,"available":96511361024,"internally_reserved":0,"allocated":93536256,"data_stored":103288934,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.76100000000000001,"15min":0.77300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90200000000000002,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.76400000000000001,"15min":0.78100000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.331,"15min":1.331},"last":0.74099999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.82899999999999996,"15min":0.84699999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.86899999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93799999999999994}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.80500000000000005,"15min":0.81100000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.101,"5min":1.298,"15min":1.298},"last":0.81899999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.04,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.497},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78100000000000003,"15min":0.78700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93500000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.70599999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82599999999999996,"15min":0.83699999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.048,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.63200000000000001}]}]},{"osd":2,"up_from":12,"seq":51539607585,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":25652,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":26267648,"data_stored":25645171,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.66700000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.70099999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97399999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.82999999999999996,"15min":0.83799999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.82399999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.093,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.88,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.89100000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95999999999999996,"15min":0.96799999999999997},"min":{"1min":0.622,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.94799999999999995,"15min":0.95499999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.85499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51449856,"data_stored":51429547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62009344,"data_stored":72482984,"data_compressed":7121,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25182208,"data_stored":25231452,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":92463104,"data_stored":102875285,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20979712,"data_stored":20971574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:17:48.214 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [1, 0] 2024-09-21T15:17:48.215 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.1', '1', '0'] 2024-09-21T15:17:48.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.1 1 0 2024-09-21T15:17:48.238 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:48.236+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:48.288 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:48.288 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:48.288 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:48.288 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:48.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:48.338 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:48.336+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:48.428 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:48.428 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:48.428 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:48.428 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:48.428 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:48.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:48.439 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:48.439+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:48.540 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:48.539+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:48.546 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 150.763354284, 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.086417061999995, 2024-09-21T15:17:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.758397693000006, 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.707236249999994, 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.619190283999998, 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.038658505000001, 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.632413823, 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.069630052999997, 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.986921279999997, 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.944802006000003, 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:48.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:48.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.798311505000001, 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:48.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:48.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.702006083000001, 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.605 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:48.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.151835611000003, 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:48.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.098997087000001, 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:174 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.053155574999998, 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033125337999999997, 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.621 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 174 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492420+0000", 2024-09-21T15:17:48.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.492419+0000", 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493270+0000", 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085079899999999996 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493286+0000", 2024-09-21T15:17:48.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6722999999999999e-05 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493294+0000", 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493355+0000", 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1459999999999998e-05 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493441+0000", 2024-09-21T15:17:48.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5877999999999993e-05 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493565+0000", 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012348200000000001 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.520406+0000", 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02684079 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525484+0000", 2024-09-21T15:17:48.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050785300000000004 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525519+0000", 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4357999999999999e-05 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525545+0000", 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6582000000000001e-05 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.221065410000001, 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:48.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:48.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.217503848, 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.631 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:48.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:48.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.213718673999999, 2024-09-21T15:17:48.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:48.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.639 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.038683832, 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:48.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:48.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.150891452, 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.644 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:48.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:17:48.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:17:48.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:48.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:48.649 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:48.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:48.651 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:48.648+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:48.751 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:48.752+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:48.764 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:48.764 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:48.764 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:48.764 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:48.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:48.852 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:48.852+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:48.886 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:48.887 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:48.887 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:48.887 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:48.887 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:48.902 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:17:48.903 INFO:teuthology.orchestra.run.smithi067.stderr:set 2.1 pg_upmap mapping to [1,0] 2024-09-21T15:17:48.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:48.953 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:48.952+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 146.184197205, 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.621772702999998, 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.611036202000001, 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:49.020 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.566579367999999, 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:49.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:49.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.558802565999997, 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:49.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.519628894999997, 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:49.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:49.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.512896533999999, 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:49.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:49.037 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.478845042000003, 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:49.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:49.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:49.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.468719057999998, 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:49.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.771955673999997, 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:49.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:49.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.634576371000001, 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:49.050 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:49.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:49.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.633752319000003, 2024-09-21T15:17:49.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.748690771, 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:49.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:49.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:49.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.747734700000002, 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:49.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.608059316999999, 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:49.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:49.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.314146041999997, 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:49.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.674207301999999, 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:17:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:49.081 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7520864820000002, 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:17:49.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:17:49.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7493601180000002, 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:49.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7467625919999996, 2024-09-21T15:17:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:49.092 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:49.092 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:49.093 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:49.093 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:49.091+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:49.194 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:49.195+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:49.211 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:49.212 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:49.212 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:49.221 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:49.295 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:49.292+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:49.327 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:49.328 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:49.328 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:49.328 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:49.328 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:49.344 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:49.396 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:49.395+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:49.450 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.824634270999994, 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:49.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:49.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.708486334, 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.661865937000002, 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:49.462 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.611249313000002, 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.463 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.522895840999993, 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.033764970999997, 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.471 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:49.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:49.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:49.475 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.318884128999997, 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:49.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:49.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.479 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.230569872000004, 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:49.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:49.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 63.848598043000003, 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:49.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.962959593999997, 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:49.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:49.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 52.111224264999997, 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:49.492 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:49.496 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.961432405, 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029737039, 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487914+0000", 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489004+0000", 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010899900000000001 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489031+0000", 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7645e-05 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489053+0000", 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489151+0000", 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7928000000000002e-05 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489165+0000", 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3451999999999999e-05 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.517652+0000", 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028487375999999998 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.8990822979999997, 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.500 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:17:49.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.636878426, 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:17:49.507 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.6038284679999997, 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:49.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:17:49.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.514971192, 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:17:49.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:17:49.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.4457177259999998, 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.519 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.4018301580000001, 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:17:49.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:17:49.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.356949449, 2024-09-21T15:17:49.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:17:49.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 5.2645188740000002, 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.527 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:17:49.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:17:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:17:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.531 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:49.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:49.533 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:49.531+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:49.633 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:49.635+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:49.647 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:49.647 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:49.647 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:49.647 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:49.656 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:49.734 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:49.735+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:49.765 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:49.765 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:49.765 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:49.765 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:49.765 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:49.774 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:49.835 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:49.835+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:49.887 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:49.887 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:49.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:49.887 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.621972992000003, 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:49.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.547747776000001, 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:49.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.895 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.537348336999997, 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:49.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:49.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.899 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.475041331, 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:49.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.412831263000001, 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:49.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:49.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:49.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.715696027, 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:49.907 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.909 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.912 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.715246049000001, 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.913 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:49.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:49.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.714996184, 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.918 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:49.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:49.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:49.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.714720679999999, 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:49.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.714317715, 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.714111678999998, 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.935 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:49.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:49.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.940 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.713907675999998, 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:49.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:49.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:49.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:49.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:49.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.713716794, 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.946 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:49.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.576653468000003, 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:17:49.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:17:49.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.576418445999998, 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:17:49.957 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:17:49.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.576218023000003, 2024-09-21T15:17:49.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:17:49.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:17:49.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.575814440999999, 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.969 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:17:49.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:17:49.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.575555284000004, 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.575360056999997, 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.980 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:17:49.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:17:49.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:17:49.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 32.381108292999997, 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:17:49.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:17:49.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:49.991 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:49.993 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:49.993 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:49.991+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:50.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:50.094 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:50.092+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:50.134 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:50.134 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:50.134 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:50.134 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:50.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:50.164 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:50.185 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:50.195 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:50.195+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:50.207 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:50.228 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:50.248 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:50.248 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:50.248 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:50.248 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:50.248 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:50.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:50.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:50.282 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:50.285 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:50.285 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:50.285 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:50.296 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:50.292+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:50.361 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:50.362 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:50.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:50.397 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:50.396+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:50.477 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:50.477 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:50.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:50.498 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:50.495+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:50.584 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:50.584 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:50.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:50.599 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:50.596+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:50.700 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:50.696+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:50.756 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:50.756 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:50.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:50.801 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:50.800+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:50.871 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:50.871 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:50.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:50.902 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:50.899+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:50.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:17:50.992 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:50.992 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:51.003 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:51.003+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:51.103 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:51.103+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:51.105 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.105 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:51.204 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:51.204+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:51.218 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.218 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:51.305 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:51.303+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:51.335 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.335 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:51.407 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:51.407+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.507 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:51.508 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:51.507+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:51.608 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:51.607+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:51.626 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.626 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.636 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:51.709 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:51.708+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:51.746 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.746 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.756 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:51.810 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:51.811+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.859 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:51.911 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:51.908+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:51.927 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:17:51.973 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:51.973 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:51.983 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:52.012 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:52.011+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:52.087 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:52.087 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:52.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:52.113 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:52.112+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:52.203 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:52.204 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:52.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:52.257 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:52.256+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:52.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:52.316 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:52.312+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:52.362 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:52.362 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:52.372 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:52.417 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:52.415+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:52.448 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:52.469 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:52.476 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:52.477 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:52.487 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:52.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:52.513 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:52.518 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:52.516+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:52.540 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:52.564 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:52.601 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:52.601 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:52.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:52.601 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.496352862999998, 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:52.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.496301819999999, 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.606 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:52.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.480682277, 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.611 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 71.480655523999999, 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:52.615 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:52.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.311607421000005, 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:52.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:52.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:52.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:52.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.296462563999995, 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.626 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:52.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.267315748000001, 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.632 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:52.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.257988843000007, 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:52.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.643 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.230352690999993, 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:52.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:52.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.219274986000002, 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:52.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:52.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:52.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.184505439000006, 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.656 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.295981357000002, 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:52.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.667 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.285726289999999, 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:52.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:52.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.253679293000005, 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.673 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:52.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:52.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:52.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.24146734, 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:52.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.207571381999998, 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:52.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:52.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.690 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.168526513000003, 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.308083386, 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.696 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:52.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:52.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:52.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:52.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:52.702 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.296505879000001, 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:52.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:52.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:52.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.262095111000001, 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:52.713 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:52.715 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:52.712+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:52.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:52.816 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:52.815+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:52.831 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:52.832 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:52.832 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:52.832 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:52.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:52.917 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:52.915+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:52.950 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:52.951 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:52.951 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:52.951 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:52.951 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:52.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:53.018 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:53.016+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:53.072 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:53.072 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:53.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.611098064999993, 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:53.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:53.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.563213808, 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:53.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.515041961999998, 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:53.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:53.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:53.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.156872859999993, 2024-09-21T15:17:53.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:53.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:53.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.990888988999998, 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.097 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:53.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.805383719999995, 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.103 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:53.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:53.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:53.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:53.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.826744888999997, 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:53.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:53.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:53.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:53.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.115 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.825960012000003, 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:53.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:53.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.825419887999999, 2024-09-21T15:17:53.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:53.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:53.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.825217471999999, 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.128 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:53.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.134 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.825021266999997, 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:53.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:53.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:53.140 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.824827143999997, 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:53.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.146 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.770379923, 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:53.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:53.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.770247476000002, 2024-09-21T15:17:53.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:53.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.770096097, 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:53.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:53.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:53.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.769968726000002, 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:53.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:53.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.170 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.635168726000003, 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.171 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:53.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:53.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.498142641999998, 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.177 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:53.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.322845737999998, 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.183 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.184 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.184 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.184 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.184 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:53.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:53.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:53.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:53.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:53.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.272553213000002, 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:53.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:53.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:53.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:53.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:53.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.197 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.197 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.197 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:53.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:53.199 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:53.199+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:53.299 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:53.299+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:53.358 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:53.358 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:53.358 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:53.358 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:53.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:53.399 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:53.399+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:53.478 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:53.479 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:53.479 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:53.479 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:53.479 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:53.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:53.531 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:53.531+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:53.602 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:53.600+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:53.642 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 155.86117771100001, 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.643 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:53.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.184240489000004, 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:53.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.856221120000001, 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:53.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:53.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:53.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.805059677000003, 2024-09-21T15:17:53.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:53.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:53.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.717013711000007, 2024-09-21T15:17:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:53.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:53.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:53.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.136481931999995, 2024-09-21T15:17:53.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:53.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.730237250000002, 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:53.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:53.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:53.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.167453480000006, 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:53.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:53.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.084744706999999, 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:53.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:53.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.042625432999998, 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:53.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.896134932000002, 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:53.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.799829510000002, 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.706 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:53.710 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.249659037999997, 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:53.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:53.716 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.196820514000002, 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:53.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.318888837000003, 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:53.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:53.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.315327275000001, 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.311542101000001, 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.731 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:53.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.136507258999998, 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:53.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:53.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.248714879000001, 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:17:53.740 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:17:53.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:17:53.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:53.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:17:53.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.1 95/94 [PushOp(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head, version: 50'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head@50'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.690927946, 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034730616999999998, 2024-09-21T15:17:53.746 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:53.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952469+0000", 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953866+0000", 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013966040000000001 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953902+0000", 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6279000000000003e-05 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953930+0000", 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7988e-05 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954032+0000", 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102122 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954046+0000", 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3784000000000001e-05 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.987201+0000", 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033155234999999998 2024-09-21T15:17:53.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.750 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:53.750 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:53.750 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:53.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:53.752 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:53.752+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:53.852 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:53.852+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:53.867 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:53.867 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:53.867 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:53.868 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:53.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:53.917 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:53.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:53.918 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T15:17:53.918 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:17:53.918 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:17:53.918 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-21T15:17:53.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:17:53.968 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:53.968+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:53.988 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:53.989 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:53.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:54.034 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:54.034 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "" 2024-09-21T15:17:54.035 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:54.054 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:54.052+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:54.104 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 151.28198618299999, 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:54.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.719561681000002, 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:54.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:54.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.708825179999998, 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:54.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:54.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:54.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.664368346000003, 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.117 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:54.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.656591544000001, 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:54.122 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:54.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.617417873000001, 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:54.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.610685512000003, 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.57663402, 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.566508036000002, 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.869744652000001, 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.732365348999998, 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.731541297, 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.846479748999997, 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.845523677999999, 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.705848295000003, 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.411935020000001, 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 28.77199628, 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:17:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8498754599999998, 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8471490960000008, 2024-09-21T15:17:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:17:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8445515700000001, 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:54.193 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:54.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:54.195 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:54.195+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:54.296 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:54.292+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:54.312 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:54.313 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:54.313 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:54.323 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:54.396 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:54.396+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:54.428 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:54.428 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:54.428 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:54.428 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:54.428 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:54.438 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:54.498 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:54.496+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:54.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.966964786000005, 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:17:54.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:17:54.598 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.850816848999997, 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:17:54.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.602 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.804196451999999, 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:54.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:17:54.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.753579827999999, 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:17:54.607 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:17:54.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.665226356000005, 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:17:54.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:17:54.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.176095485999994, 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.461214643999995, 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:54.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:17:54.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:54.624 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.372900387000001, 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:17:54.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:17:54.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.628 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.990928557999993, 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.105290109000002, 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:17:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 57.253554780000002, 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:17:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:17:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:17:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.103762919999999, 2024-09-21T15:17:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029737039, 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487914+0000", 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489004+0000", 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010899900000000001 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489031+0000", 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7645e-05 2024-09-21T15:17:54.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489053+0000", 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489151+0000", 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7928000000000002e-05 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489165+0000", 2024-09-21T15:17:54.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3451999999999999e-05 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.517652+0000", 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028487375999999998 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 11.041412813000001, 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:17:54.645 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:17:54.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:17:54.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.779208941, 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:17:54.649 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.746158983000001, 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.653 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:17:54.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.657301707, 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:54.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:17:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:17:54.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.588048240999999, 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:54.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:17:54.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:17:54.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.544160673, 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:54.668 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.499279963999999, 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:17:54.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:17:54.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 10.406849389, 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:54.672 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:17:54.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:17:54.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:17:54.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:17:54.676 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.676 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.676 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.676 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:54.676 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:54.676 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:54.677 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:54.678 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:54.675+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:54.732 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:54.753 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:54.773 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:54.778 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:54.775+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:54.789 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:54.789 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:54.795 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:54.800 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:54.816 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:54.839 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:54.878 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:54.879+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:54.904 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:54.904 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:54.915 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:54.979 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:54.979+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:55.019 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.019 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.029 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:17:55.080 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:55.076+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:55.131 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.131 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.141 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:55.180 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:55.179+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:55.281 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:55.280+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:55.293 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.293 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.303 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:55.382 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:55.380+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:55.417 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.417 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:55.483 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:55.480+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:55.542 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.542 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:55.598 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:55.599+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:55.664 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.664 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:55.684 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:55.684+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:55.780 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.780 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.785 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:55.784+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:55.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:55.886 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:55.884+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:55.900 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:55.900 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:55.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:55.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:55.987+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:56.015 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.015 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:56.088 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:56.084+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:56.188 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:56.187+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:56.245 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.245 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:56.289 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:56.288+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:56.390 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:56.391+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:56.412 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.412 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:56.491 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:56.491+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:56.531 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.531 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:56.592 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:56.591+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:56.662 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.662 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.672 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:56.692 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:56.691+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:56.779 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.779 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.789 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:56.793 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:56.792+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:56.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:17:56.893 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:56.892+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:56.898 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:56.898 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:56.908 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:56.995 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:56.995+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:57.008 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:57.032 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:57.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:17:57.054 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:57.063 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:57.063 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:57.073 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:57.090 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:57.095 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:57.095+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:57.117 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:57.137 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:57.140 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:57.140 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:57.140 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:57.140 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:57.140 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:57.140 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:57.177 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:57.178 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:57.178 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:57.178 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:57.178 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:57.188 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:17:57.196 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:57.192+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:57.295 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 54.030163610000002, 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:17:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:17:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.955938394, 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:17:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:17:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.945538955000004, 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.883231948999999, 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:57.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:17:57.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.311 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 53.821021881, 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:17:57.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.315 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.123886644999999, 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:57.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:17:57.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:17:57.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:17:57.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.123436667, 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.321 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.326 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.123186801999999, 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:17:57.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.122911297999998, 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.332 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:17:57.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.337 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.122508332999999, 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.338 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:17:57.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:17:57.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:57.343 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.122302296999997, 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:17:57.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:17:57.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.122098293999997, 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.349 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:17:57.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:17:57.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 41.121907411999999, 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.355 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:17:57.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.360 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.984844086000003, 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:17:57.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.984609063999997, 2024-09-21T15:17:57.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:17:57.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:17:57.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:17:57.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.984408641000002, 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.372 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:17:57.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.377 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.984005058999998, 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:17:57.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:17:57.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:57.383 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.983745902000003, 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:17:57.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.983550675000004, 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.389 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:17:57.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:17:57.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:17:57.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 39.789298911000003, 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:57.395 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:17:57.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.400 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:57.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:17:57.402 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:57.403+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:57.502 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:57.500+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:57.558 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:57.558 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:57.558 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:57.559 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:57.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:17:57.603 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:57.603+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:57.672 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:57.672 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:57.672 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:17:57.673 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:17:57.673 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:57.682 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:17:57.704 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:57.703+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:57.789 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.683630292000004, 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:57.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:17:57.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:17:57.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.793 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.683579249000005, 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:17:57.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.667959706000005, 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.798 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:57.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.802 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.667932953000005, 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:17:57.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:17:57.806 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 73.498884849999996, 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.807 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:17:57.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:57.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:17:57.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:17:57.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.813 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 73.483739993, 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:17:57.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:57.819 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 73.454593177000007, 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.824 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 73.445266271999998, 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:17:57.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.830 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 73.417630119999998, 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.831 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.836 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 73.406552414999993, 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.837 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:17:57.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:17:57.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.842 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 73.371782867999997, 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.848 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.483258785999993, 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.849 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:17:57.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:17:57.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.854 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.473003719000005, 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:17:57.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:17:57.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:17:57.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:17:57.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.440956721999996, 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:17:57.860 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:17:57.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:17:57.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:17:57.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.428744769000005, 2024-09-21T15:17:57.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:57.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.868 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.869 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:17:57.870 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:17:57.871 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.872 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.394848811000003, 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:57.873 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.874 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.875 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.876 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.877 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:57.878 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.355803941999994, 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:17:57.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:17:57.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.884 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.495360814999998, 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.885 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:17:57.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:17:57.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:17:57.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.483783308, 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:57.892 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:17:57.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:17:57.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 60.449372539999999, 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:17:57.898 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:57.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:17:57.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:17:57.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:17:57.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:17:57.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:17:57.904 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:17:57.905 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:57.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:17:57.906 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:57.907+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:58.007 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:58.004+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:58.025 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:58.026 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:58.026 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:58.026 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:58.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:17:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:17:58.108 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:58.104+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:58.136 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:58.136 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:58.136 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:58.136 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:58.136 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:58.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:17:58.208 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:58.208+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.791393069999998, 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:17:58.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:17:58.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:17:58.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.743508813000005, 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.255 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:17:58.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:17:58.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:17:58.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.695336967000003, 2024-09-21T15:17:58.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:17:58.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.337167864999998, 2024-09-21T15:17:58.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:17:58.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:17:58.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.171183994000003, 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.272 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:17:58.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:17:58.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:17:58.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.985678725, 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:58.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:17:58.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:17:58.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.007039894000002, 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.283 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.288 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.006255017000001, 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:17:58.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:58.294 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.005714893000004, 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:58.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.005512476999996, 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:17:58.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:17:58.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:17:58.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.005316272000002, 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.005122149000002, 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.311 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:17:58.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:17:58.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.950674927999998, 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:17:58.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.950542480999999, 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:17:58.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.950391101999998, 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.328 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.950263731, 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:17:58.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:58.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:17:58.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.815463731000001, 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.338 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.678437647000003, 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:17:58.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.503140743000003, 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:17:58.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:58.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:17:58.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.354 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.452848218, 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:17:58.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.359 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:58.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:17:58.361 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:58.360+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:58.461 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:58.459+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:58.476 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:58.476 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:58.476 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:58.476 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:58.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:17:58.562 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:58.560+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:58.596 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:58.596 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:58.596 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:58.596 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:58.596 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:58.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:17:58.663 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:58.664+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:58.714 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:58.714 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:58.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:58.714 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:58.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.714 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 160.93262169400001, 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:17:58.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:17:58.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:17:58.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.255684471999999, 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:17:58.720 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:58.724 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.927665102999995, 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:17:58.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:17:58.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:17:58.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.729 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.876503659999997, 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.730 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:17:58.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:17:58.735 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.788457694000002, 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:58.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:17:58.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.207925915000004, 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:17:58.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.801681232999996, 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:17:58.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.238897463000001, 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:17:58.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.753 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.156188689999993, 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.114069416000007, 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:17:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:17:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.967578914999997, 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:17:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:58.768 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.871273492999997, 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.321103020999999, 2024-09-21T15:17:58.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:17:58.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:17:58.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.268264496999997, 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.778 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:17:58.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:17:58.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:17:58.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.783 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.390332819999998, 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:17:58.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:17:58.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.386771258000003, 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:17:58.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.791 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.382986084000002, 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.207951242, 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:17:58.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:17:58.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.320158862, 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:58.800 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:17:58.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.1 95/94 [PushOp(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head, version: 50'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head@50'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:17:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7623719290000004, 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034730616999999998, 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:58.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952469+0000", 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953866+0000", 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013966040000000001 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953902+0000", 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6279000000000003e-05 2024-09-21T15:17:58.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953930+0000", 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7988e-05 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954032+0000", 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102122 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954046+0000", 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3784000000000001e-05 2024-09-21T15:17:58.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.987201+0000", 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033155234999999998 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:58.809 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:58.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:17:58.811 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:58.808+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:58.911 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:17:58.911+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:17:58.921 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:17:58.921 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:58.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:17:59.015 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:59.015+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:59.035 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:59.035 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:17:59.035 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:17:59.035 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:17:59.035 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:59.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:17:59.045 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:17:59.046 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:17:59.046 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T15:17:59.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:17:59.116 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:59.115+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:59.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:17:59.153 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 156.33170231599999, 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:17:59.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.769277813999999, 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:17:59.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.758541313000002, 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.163 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:17:59.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:17:59.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.714084479, 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:17:59.167 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.706307676999998, 2024-09-21T15:17:59.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:17:59.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.667134005999998, 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.179 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.660401645, 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:59.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.626350152999997, 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.184 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:17:59.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.188 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.616224168999999, 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:17:59.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:17:59.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.919460784999998, 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:17:59.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:17:59.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.782081482000002, 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:17:59.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:17:59.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.781257429999997, 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:17:59.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.896195882000001, 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.206 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:17:59.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:17:59.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.895239811000003, 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:17:59.211 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.755564428, 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:59.215 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:17:59.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.219 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.461651152999998, 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:59.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:17:59.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.821712413, 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:17:59.224 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:17:59.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:17:59.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.899591593, 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:17:59.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.896865228999999, 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:17:59.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.894267703000001, 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:17:59.237 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:17:59.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.240 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:59.242 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:17:59.243 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:17:59.243+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 98, 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:17:58.026305+0000", 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:17:42.447038+0000", 2024-09-21T15:17:59.342 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5801984, 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:17:59.343 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:17:59.344 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:17:59.345 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:59.346 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:17:59.347 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:17:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:59.349 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:59.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:17:59.351 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:59.352 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:59.353 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:59.354 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "87", 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 8, 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 87, 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:59.357 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 3, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 3, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 3, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 3, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:59.358 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "new_default_pool17425", 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:16:34.640599+0000", 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:17:59.359 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:17:59.360 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "54", 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 54, 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:17:59.361 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:17:59.362 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:17:59.363 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 2.2699999809265137, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 2.2699999809265137, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.364 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 68, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:59.365 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:17:59.369 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 94, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:59.375 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 91, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:59.376 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.377 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.378 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:59.379 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:17:59.380 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:17:59.381 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:17:59.384 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:17:59.385 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:59.386 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:59.387 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 91, 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:17:59.388 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.389 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.390 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:17:59.391 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:59.392 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:17:59.393 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.395 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:17:59.396 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:17:59.397 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:59.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:59.399 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:17:59.400 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:17:59.401 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.1", 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: 0 2024-09-21T15:17:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.1", 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: 2, 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: 3 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.4", 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-09-21T15:17:59.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 4 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:17:59.404 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:17:59.405 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:17:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:17:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:17:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:17:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:17:59.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:17:59.406 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:17:59.408 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:17:59.408 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:17:59.408 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:17:59.408 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:17:59.409 INFO:tasks.thrashosds.thrasher:j is {'epoch': 98, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:17:58.026305+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:17:42.447038+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'new_default_pool17425', 'create_time': '2024-09-21T15:16:34.640599+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 68, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.1', 'osds': [1, 0]}, {'pgid': '3.1', 'osds': [2, 3]}], 'pg_upmap_items': [{'pgid': '2.4', 'mappings': [{'from': 5, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:17:59.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:17:59.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:17:59.410 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:17:59.404+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:17:59.410 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:17:59.410 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:17:59.426 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:17:59.449 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:17:59.471 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:17:59.494 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:17:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:17:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:17:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:17:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:17:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:17:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:17:59.508 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:59.504+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:59.524 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:59.524 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:59.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:17:59.608 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:59.608+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:59.690 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:59.691 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:59.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:17:59.712 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:17:59.714 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:17:59.726 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-21T15:17:58.034218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147465123,"num_objects":77,"num_object_clones":19,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":13,"num_read":5722,"num_read_kb":155475,"num_write":936,"num_write_kb":273253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":167772427,"num_keys_recovered":45,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":890,"ondisk_log_size":890,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":464212,"kb_used_data":277512,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":565766828,"statfs":{"total":579820584960,"available":579345231872,"internally_reserved":0,"allocated":284172288,"data_stored":301960442,"data_compressed":25932,"data_compressed_allocated":368640,"data_compressed_original":9846784,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.115128"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":242,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.924505+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:17:51.924505+0000","last_peered":"2024-09-21T15:17:51.924505+0000","last_clean":"2024-09-21T15:17:51.924505+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:17:51.924505+0000","last_undegraded":"2024-09-21T15:17:51.924505+0000","last_fullsized":"2024-09-21T15:17:51.924505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'45","reported_seq":1138,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.924553+0000","last_change":"2024-09-21T15:16:55.672969+0000","last_active":"2024-09-21T15:17:51.924553+0000","last_peered":"2024-09-21T15:17:51.924553+0000","last_clean":"2024-09-21T15:17:51.924553+0000","last_became_active":"2024-09-21T15:16:55.272135+0000","last_became_peered":"2024-09-21T15:16:55.272135+0000","last_unstale":"2024-09-21T15:17:51.924553+0000","last_undegraded":"2024-09-21T15:17:51.924553+0000","last_fullsized":"2024-09-21T15:17:51.924553+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:58.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":20971558,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":960,"num_read_kb":31831,"num_write":42,"num_write_kb":32833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"75'6","reported_seq":95,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:50.056413+0000","last_change":"2024-09-21T15:17:44.260941+0000","last_active":"2024-09-21T15:17:50.056413+0000","last_peered":"2024-09-21T15:17:50.056413+0000","last_clean":"2024-09-21T15:17:50.056413+0000","last_became_active":"2024-09-21T15:17:43.480798+0000","last_became_peered":"2024-09-21T15:17:43.480798+0000","last_unstale":"2024-09-21T15:17:50.056413+0000","last_undegraded":"2024-09-21T15:17:50.056413+0000","last_fullsized":"2024-09-21T15:17:50.056413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:07.595836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181755,"stat_sum":{"num_bytes":54,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":127,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"82'5","reported_seq":67,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:50.056441+0000","last_change":"2024-09-21T15:16:55.569580+0000","last_active":"2024-09-21T15:17:50.056441+0000","last_peered":"2024-09-21T15:17:50.056441+0000","last_clean":"2024-09-21T15:17:50.056441+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:17:50.056441+0000","last_undegraded":"2024-09-21T15:17:50.056441+0000","last_fullsized":"2024-09-21T15:17:50.056441+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:50.056660+0000","last_change":"2024-09-21T15:17:43.481624+0000","last_active":"2024-09-21T15:17:50.056660+0000","last_peered":"2024-09-21T15:17:50.056660+0000","last_clean":"2024-09-21T15:17:50.056660+0000","last_became_active":"2024-09-21T15:17:43.481181+0000","last_became_peered":"2024-09-21T15:17:43.481181+0000","last_unstale":"2024-09-21T15:17:50.056660+0000","last_undegraded":"2024-09-21T15:17:50.056660+0000","last_fullsized":"2024-09-21T15:17:50.056660+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:03:25.583980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"85'22","reported_seq":27,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.924553+0000","last_change":"2024-09-21T15:17:50.742335+0000","last_active":"2024-09-21T15:17:51.924553+0000","last_peered":"2024-09-21T15:17:51.924553+0000","last_clean":"2024-09-21T15:17:51.924553+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:17:51.924553+0000","last_undegraded":"2024-09-21T15:17:51.924553+0000","last_fullsized":"2024-09-21T15:17:51.924553+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":35,"num_read_kb":16384,"num_write":21,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":66,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.922047+0000","last_change":"2024-09-21T15:16:55.272512+0000","last_active":"2024-09-21T15:17:51.922047+0000","last_peered":"2024-09-21T15:17:51.922047+0000","last_clean":"2024-09-21T15:17:51.922047+0000","last_became_active":"2024-09-21T15:16:55.271874+0000","last_became_peered":"2024-09-21T15:16:55.271874+0000","last_unstale":"2024-09-21T15:17:51.922047+0000","last_undegraded":"2024-09-21T15:17:51.922047+0000","last_fullsized":"2024-09-21T15:17:51.922047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:54:48.789474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046520300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"87'25","reported_seq":282,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.924563+0000","last_change":"2024-09-21T15:17:01.768225+0000","last_active":"2024-09-21T15:17:51.924563+0000","last_peered":"2024-09-21T15:17:51.924563+0000","last_clean":"2024-09-21T15:17:51.924563+0000","last_became_active":"2024-09-21T15:17:01.327838+0000","last_became_peered":"2024-09-21T15:17:01.327838+0000","last_unstale":"2024-09-21T15:17:51.924563+0000","last_undegraded":"2024-09-21T15:17:51.924563+0000","last_fullsized":"2024-09-21T15:17:51.924563+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:08.047427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":12582969,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":132,"num_read_kb":8306,"num_write":24,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'1","reported_seq":71,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.924528+0000","last_change":"2024-09-21T15:16:55.676403+0000","last_active":"2024-09-21T15:17:51.924528+0000","last_peered":"2024-09-21T15:17:51.924528+0000","last_clean":"2024-09-21T15:17:51.924528+0000","last_became_active":"2024-09-21T15:16:55.274451+0000","last_became_peered":"2024-09-21T15:16:55.274451+0000","last_unstale":"2024-09-21T15:17:51.924528+0000","last_undegraded":"2024-09-21T15:17:51.924528+0000","last_fullsized":"2024-09-21T15:17:51.924528+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:20.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046455,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"75'1","reported_seq":69,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:50.056703+0000","last_change":"2024-09-21T15:17:44.264959+0000","last_active":"2024-09-21T15:17:50.056703+0000","last_peered":"2024-09-21T15:17:50.056703+0000","last_clean":"2024-09-21T15:17:50.056703+0000","last_became_active":"2024-09-21T15:17:43.481232+0000","last_became_peered":"2024-09-21T15:17:43.481232+0000","last_unstale":"2024-09-21T15:17:50.056703+0000","last_undegraded":"2024-09-21T15:17:50.056703+0000","last_fullsized":"2024-09-21T15:17:50.056703+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:11:58.858880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":32,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"85'27","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:49.906542+0000","last_change":"2024-09-21T15:17:44.249295+0000","last_active":"2024-09-21T15:17:49.906542+0000","last_peered":"2024-09-21T15:17:49.906542+0000","last_clean":"2024-09-21T15:17:49.906542+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:17:49.906542+0000","last_undegraded":"2024-09-21T15:17:49.906542+0000","last_fullsized":"2024-09-21T15:17:49.906542+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":52,"num_read_kb":8203,"num_write":26,"num_write_kb":32770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"75'1","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:49.906493+0000","last_change":"2024-09-21T15:17:01.318184+0000","last_active":"2024-09-21T15:17:49.906493+0000","last_peered":"2024-09-21T15:17:49.906493+0000","last_clean":"2024-09-21T15:17:49.906493+0000","last_became_active":"2024-09-21T15:17:01.317913+0000","last_became_peered":"2024-09-21T15:17:01.317913+0000","last_unstale":"2024-09-21T15:17:49.906493+0000","last_undegraded":"2024-09-21T15:17:49.906493+0000","last_fullsized":"2024-09-21T15:17:49.906493+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:47:53.254684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043867299999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"89'174","reported_seq":1333,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.924630+0000","last_change":"2024-09-21T15:17:24.302020+0000","last_active":"2024-09-21T15:17:51.924630+0000","last_peered":"2024-09-21T15:17:51.924630+0000","last_clean":"2024-09-21T15:17:51.924630+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:17:51.924630+0000","last_undegraded":"2024-09-21T15:17:51.924630+0000","last_fullsized":"2024-09-21T15:17:51.924630+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":27287606,"num_objects":14,"num_object_clones":6,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":1248,"num_read_kb":38813,"num_write":192,"num_write_kb":52364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":56,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:49.906597+0000","last_change":"2024-09-21T15:17:43.480003+0000","last_active":"2024-09-21T15:17:49.906597+0000","last_peered":"2024-09-21T15:17:49.906597+0000","last_clean":"2024-09-21T15:17:49.906597+0000","last_became_active":"2024-09-21T15:17:43.479431+0000","last_became_peered":"2024-09-21T15:17:43.479431+0000","last_unstale":"2024-09-21T15:17:49.906597+0000","last_undegraded":"2024-09-21T15:17:49.906597+0000","last_fullsized":"2024-09-21T15:17:49.906597+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:19.528788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092366200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"89'244","reported_seq":1251,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:52.357549+0000","last_change":"2024-09-21T15:17:05.211552+0000","last_active":"2024-09-21T15:17:52.357549+0000","last_peered":"2024-09-21T15:17:52.357549+0000","last_clean":"2024-09-21T15:17:52.357549+0000","last_became_active":"2024-09-21T15:17:05.209676+0000","last_became_peered":"2024-09-21T15:17:05.209676+0000","last_unstale":"2024-09-21T15:17:52.357549+0000","last_undegraded":"2024-09-21T15:17:52.357549+0000","last_fullsized":"2024-09-21T15:17:52.357549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:02:52.206876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18923558,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":2396,"num_read_kb":34834,"num_write":141,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":20,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":56,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T15:17:49.906724+0000","last_change":"2024-09-21T15:17:43.476863+0000","last_active":"2024-09-21T15:17:49.906724+0000","last_peered":"2024-09-21T15:17:49.906724+0000","last_clean":"2024-09-21T15:17:49.906724+0000","last_became_active":"2024-09-21T15:17:43.476485+0000","last_became_peered":"2024-09-21T15:17:43.476485+0000","last_unstale":"2024-09-21T15:17:49.906724+0000","last_undegraded":"2024-09-21T15:17:49.906724+0000","last_fullsized":"2024-09-21T15:17:49.906724+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:37.900423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"89'222","reported_seq":481,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-21T15:17:51.924798+0000","last_change":"2024-09-21T15:17:24.303933+0000","last_active":"2024-09-21T15:17:51.924798+0000","last_peered":"2024-09-21T15:17:51.924798+0000","last_clean":"2024-09-21T15:17:51.924798+0000","last_became_active":"2024-09-21T15:17:24.303534+0000","last_became_peered":"2024-09-21T15:17:24.303534+0000","last_unstale":"2024-09-21T15:17:51.924798+0000","last_undegraded":"2024-09-21T15:17:51.924798+0000","last_fullsized":"2024-09-21T15:17:51.924798+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:39.035665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":20971596,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":4,"num_read":719,"num_read_kb":16966,"num_write":321,"num_write_kb":53462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":159,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146874685,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":13,"num_read":5568,"num_read_kb":155345,"num_write":771,"num_write_kb":270831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":167772268,"num_keys_recovered":29,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277250048,"data_stored":298156666,"data_compressed":14242,"data_compressed_allocated":294912,"data_compressed_original":8699904,"omap_allocated":0,"internal_metadata":0},"log_size":764,"ondisk_log_size":764,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40480,"kb_used_data":9364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94331360,"statfs":{"total":96636764160,"available":96595312640,"internally_reserved":0,"allocated":9588736,"data_stored":9417856,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.53000000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.133,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.032,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.82699999999999996,"15min":0.83699999999999997},"min":{"1min":0.53100000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2070000000000001,"5min":1.212,"15min":1.212},"last":0.69999999999999996}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65300000000000002,"15min":0.64500000000000002},"min":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84699999999999998,"15min":0.84999999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0880000000000001,"5min":1.208,"15min":1.208},"last":0.67200000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.82399999999999995,"15min":0.83599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0089999999999999,"5min":1.629,"15min":1.629},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.57899999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56816,"kb_used_data":25696,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315024,"statfs":{"total":96636764160,"available":96578584576,"internally_reserved":0,"allocated":26312704,"data_stored":25604720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.99099999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.83099999999999996},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96199999999999997,"5min":1.333,"15min":1.333},"last":0.86699999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.115,"5min":1.133,"15min":1.133},"last":0.53100000000000003}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.64200000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.84299999999999997,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.038,"5min":1.581,"15min":1.581},"last":0.96399999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.75900000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81299999999999994,"15min":0.81499999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.872,"5min":0.86599999999999999,"15min":0.86499999999999999},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.99199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105124,"kb_used_data":74008,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94266716,"statfs":{"total":96636764160,"available":96529117184,"internally_reserved":0,"allocated":75784192,"data_stored":86095090,"data_compressed":12966,"data_compressed_allocated":184320,"data_compressed_original":4923392,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.42899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.53300000000000003}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.64200000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.90300000000000002,"5min":1.226,"15min":1.226},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.016}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84699999999999998,"15min":0.84599999999999997},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.272,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86499999999999999,"15min":0.85999999999999999},"min":{"1min":0.64900000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.90700000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":1.2589999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.94899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94756,"kb_used_data":63640,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94277084,"statfs":{"total":96636764160,"available":96539734016,"internally_reserved":0,"allocated":65167360,"data_stored":64451285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57799999999999996,"15min":0.59199999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.69699999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.087},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.88300000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95499999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1.3160000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91300000000000003,"5min":1.101,"15min":1.101},"last":0.999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.90100000000000002,"15min":0.91500000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.085,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.2370000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.90400000000000003,"15min":0.92200000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0289999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.163},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.91900000000000004,"15min":0.92900000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1839999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.194}]}]},{"osd":4,"up_from":12,"seq":51539607589,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110220,"kb_used_data":79104,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94261620,"statfs":{"total":96636764160,"available":96523898880,"internally_reserved":0,"allocated":81002496,"data_stored":90726171,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.76100000000000001,"15min":0.77300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90200000000000002,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.76400000000000001,"15min":0.78100000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.331,"15min":1.331},"last":0.71799999999999997}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.82899999999999996,"15min":0.84699999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.86899999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.59499999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.80500000000000005,"15min":0.81100000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.101,"5min":1.298,"15min":1.298},"last":0.627}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.04,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.88},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78100000000000003,"15min":0.78700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93500000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.67600000000000005}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82599999999999996,"15min":0.83699999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.048,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.75700000000000001}]}]},{"osd":2,"up_from":12,"seq":51539607587,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56816,"kb_used_data":25700,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315024,"statfs":{"total":96636764160,"available":96578584576,"internally_reserved":0,"allocated":26316800,"data_stored":25665320,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.78300000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.70099999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97399999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.68799999999999994}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.82999999999999996,"15min":0.83799999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.72499999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.093,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.88,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.88}]},{"osd":5,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95999999999999996,"15min":0.96799999999999997},"min":{"1min":0.622,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.94799999999999995,"15min":0.95499999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.76000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":64032768,"data_stored":64012459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74592256,"data_stored":85065896,"data_compressed":7121,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25182208,"data_stored":25231452,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":79880192,"data_stored":90292373,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:17:59.727 INFO:tasks.thrashosds.thrasher:Setting 3.5 to [4, 1] 2024-09-21T15:17:59.728 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.5', '4', '1'] 2024-09-21T15:17:59.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.5 4 1 2024-09-21T15:17:59.744 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:17:59.744+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:17:59.809 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:59.809 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:59.809 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:17:59.807+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:17:59.818 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:17:59.911 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:17:59.908+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:17:59.928 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:17:59.928 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:17:59.939 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:00.012 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:00.008+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:00.047 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.047 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:18:00.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:00.112 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:00.111+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:00.208 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.209 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.213 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:00.211+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:00.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:00.314 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:00.312+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:00.328 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.328 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.338 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:00.415 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:00.412+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:00.442 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.442 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:00.516 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:00.512+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:00.565 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.565 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:00.617 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:00.616+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:00.684 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.684 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:00.718 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:00.716+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:00.803 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.803 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:00.818 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:00.819+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:00.919 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:00.919 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:00.920 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:00.916+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:00.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:01.021 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:01.019+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:01.037 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:01.037 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:01.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:18:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.5 pg_upmap mapping to [4,1] 2024-09-21T15:18:01.122 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:01.120+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:01.157 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:01.157 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:01.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:01.222 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:01.223+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:01.273 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:01.273 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:01.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:01.323 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:01.323+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:01.390 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:01.390 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:01.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:01.424 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:01.424+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:01.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:18:01.513 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:01.513 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:01.525 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:01.525 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:01.524+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:01.625 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:01.623+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:01.641 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:01.641 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:01.650 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:01.651 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:01.671 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:01.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:01.716 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:01.726 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:01.724+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:01.738 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:01.756 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:01.756 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:01.756 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:01.757 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:01.757 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:01.763 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:01.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:01.827 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:01.827+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.248739698999998, 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:18:01.879 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.880 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:18:01.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.882 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.883 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.132591762000004, 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:01.884 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.885 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.887 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.085971365000006, 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.888 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:18:01.889 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:18:01.890 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:18:01.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.035354741000006, 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:18:01.893 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:18:01.894 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.895 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.947001268999998, 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:18:01.897 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.898 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.899 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.457870399000001, 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:18:01.900 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.901 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:18:01.902 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:18:01.903 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.904 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.742989557000001, 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.905 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.907 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.908 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.654675299999994, 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:18:01.911 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:18:01.912 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 76.272703471, 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.913 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:01.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:18:01.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:18:01.916 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.387065022000002, 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.917 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.918 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:18:01.919 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:18:01.920 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.535329692999994, 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:18:01.921 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:18:01.922 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:18:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:18:01.924 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.385537833000001, 2024-09-21T15:18:01.925 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029737039, 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487914+0000", 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489004+0000", 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010899900000000001 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489031+0000", 2024-09-21T15:18:01.927 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7645e-05 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489053+0000", 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489151+0000", 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7928000000000002e-05 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.928 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489165+0000", 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3451999999999999e-05 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.517652+0000", 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028487375999999998 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:01.929 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.323187726, 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:01.930 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:18:01.931 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.932 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:01.933 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.060983854, 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.934 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.935 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:18:01.936 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 18.027933896, 2024-09-21T15:18:01.937 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.938 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.939 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:18:01.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.939076620000002, 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:18:01.941 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.942 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.869823153999999, 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.945 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.825935586, 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.949 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:18:01.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:18:01.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.781054876999999, 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 17.688624302000001, 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:01.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:18:01.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:18:01.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:01.960 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:01.962 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:01.962 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:01.960+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:02.062 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:02.060+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:02.120 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:02.120 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:02.121 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:02.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:02.163 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:02.160+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:02.233 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:02.317 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:02.318 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:02.319 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:02.319 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:02.322 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:02.318+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:02.323 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:02.421 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:02.420+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:02.480 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:02.480 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:02.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:02.480 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.214300156, 2024-09-21T15:18:02.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:18:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:18:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:18:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.140074939999998, 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:18:02.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:18:02.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.129675501000001, 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.496 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:18:02.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.500 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.067368494999997, 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:18:02.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 59.005158426999998, 2024-09-21T15:18:02.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:02.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:18:02.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.509 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.308023190999997, 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.510 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:18:02.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:18:02.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:18:02.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:18:02.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.307573212999998, 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.517 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:18:02.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.307323347999997, 2024-09-21T15:18:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:18:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:18:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.307047844000003, 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:18:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:18:02.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:18:02.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:18:02.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.306644878999997, 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.538 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:18:02.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:18:02.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.306438843000002, 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:18:02.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.306234840000002, 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.552 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:18:02.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.306043957999997, 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.559 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:02.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:18:02.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:18:02.565 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.168980632, 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:18:02.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.168745610000002, 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:18:02.573 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:18:02.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.579 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.168545186999999, 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:18:02.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:18:02.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.168141605000002, 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:18:02.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:18:02.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:18:02.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:18:02.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.167882448, 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.594 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:18:02.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.600 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.167687221000001, 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.601 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:18:02.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:18:02.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 44.973435457000001, 2024-09-21T15:18:02.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:18:02.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:18:02.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.615 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:02.617 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:02.617 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:02.614+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:02.718 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:02.718+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:02.729 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:02.729 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:02.729 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:02.730 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:02.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:02.819 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:02.818+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:02.848 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:02.848 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:02.848 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:02.848 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:02.848 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:02.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:02.920 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:02.920+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.863301273000005, 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.967 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:02.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:18:02.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:18:02.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:02.971 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.863250230000006, 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:18:02.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.847630687000006, 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.976 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:18:02.979 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.847603934000006, 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:18:02.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:18:02.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.982 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.678555830999997, 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.984 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:18:02.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:18:02.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:18:02.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.663410974000001, 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.990 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:18:02.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:18:02.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.634264157999993, 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:02.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:02.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:18:02.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:18:03.000 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.624937252999999, 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.001 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:18:03.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.597301100999999, 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.006 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:18:03.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.011 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.586223395999994, 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.012 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:18:03.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.551453848999998, 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.662929766999994, 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.023 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:03.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:18:03.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.652674700000006, 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.034 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.620627702999997, 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:03.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:18:03.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.608415750000006, 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.040 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:18:03.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:18:03.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:18:03.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.574519792000004, 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:18:03.045 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:18:03.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.535474922999995, 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:18:03.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.052 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:18:03.053 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.054 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.055 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.675031795999999, 2024-09-21T15:18:03.056 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.057 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.058 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.059 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.060 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.061 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:03.062 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.663454289000001, 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:18:03.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:18:03.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 65.629043521, 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:03.068 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:03.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.073 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.074 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.074 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:03.074 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:03.074 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:03.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:18:03.076 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:03.072+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:03.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:03.176 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:03.176+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:03.190 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:03.190 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:03.190 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:03.190 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:03.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:03.276 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:03.274+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:03.303 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:03.304 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:03.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:03.377 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:03.376+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:03.426 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:03.426 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.967519620999994, 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:18:03.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.432 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.919635364000001, 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.433 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:18:03.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.438 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.871463517999999, 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:18:03.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:18:03.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:18:03.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.513294415999994, 2024-09-21T15:18:03.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.449 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.347310544999999, 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.450 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:18:03.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 78.161805275999996, 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:03.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:18:03.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.183166444999998, 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:18:03.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:18:03.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:18:03.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:03.467 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.182381567999997, 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:18:03.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:18:03.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.181841444, 2024-09-21T15:18:03.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:18:03.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:18:03.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.181639028000006, 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.479 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.181442822999998, 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:18:03.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:03.490 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.181248699999998, 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:18:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:18:03.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.126801478999994, 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:18:03.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:18:03.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.501 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.126669031999995, 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:18:03.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.126517652999993, 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:18:03.507 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:18:03.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.513 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.126390282000003, 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.991590282000004, 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.519 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:03.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.524 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.854564198000006, 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:18:03.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:18:03.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:18:03.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.679267293999999, 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:18:03.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:18:03.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:18:03.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.628974768999996, 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:03.535 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:18:03.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:03.540 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:03.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:03.542 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:03.540+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:03.644 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:03.642+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:03.657 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:03.657 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:03.657 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:03.658 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:03.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:03.745 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:03.744+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:03.772 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:03.772 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:03.772 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:03.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:18:03.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:03.845 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:03.842+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:03.916 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:03.937 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:03.937 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:03.937 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:03.946 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:03.946+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:03.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:03.954 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:03.975 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:03.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:04.026 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:04.065 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.065 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.066 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:04.062+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:18:04.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:04.148 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:04.148+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:04.239 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.240 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:04.250 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:04.248+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:04.350 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:04.348+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:04.356 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.357 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:04.451 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:04.450+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:04.482 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.482 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.492 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:04.552 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:04.554+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:04.599 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.599 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.609 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:04.653 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:04.650+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:04.717 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.717 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:04.755 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:04.754+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:04.831 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.831 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.840 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:04.888 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:04.888+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:04.944 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:04.944 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:04.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:04.956 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:04.954+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:05.056 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:05.056+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:05.063 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.063 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.073 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:05.158 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:05.158+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:05.177 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.177 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:05.259 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:05.258+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:05.290 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.290 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.300 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:05.360 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:05.358+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:05.411 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.411 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:05.460 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:05.460+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:05.541 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.542 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:05.561 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:05.558+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:05.663 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:05.662+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:05.666 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.666 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:05.764 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:05.764+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:05.782 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.782 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:05.865 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:05.864+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:05.901 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:05.902 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:05.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:05.967 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:05.966+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:06.018 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:06.018 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:06.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:06.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:18:06.068 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:06.064+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:06.074 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:18:06.074 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:18:06.074 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T15:18:06.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:18:06.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:18:06.169 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:06.166+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:06.196 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:06.196 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:06.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:06.252 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:06.269 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:06.266+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:06.293 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:06.316 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:06.353 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:06.370 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:06.368+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:06.373 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:06.373 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:06.373 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:06.373 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:06.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:06.388 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:06.410 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:06.412 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:06.412 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:06.412 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:06.412 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 101, 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fa0b2d62-c633-4668-b5db-f1f6328d1af9", 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-09-21T15:15:04.255598+0000", 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-09-21T15:18:04.060642+0000", 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-09-21T15:15:15.737314+0000", 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-09-21T15:17:42.447038+0000", 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5804032, 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-09-21T15:18:06.447 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 5, 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 6, 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:18:06.448 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "reef", 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:15.995966+0000", 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:18:06.449 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:18:06.450 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:18:06.451 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:18:06.452 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:18:06.453 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.059999942779541, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.059999942779541, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T15:18:06.454 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "rbd", 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:15:21.158896+0000", 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:18:06.455 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:18:06.456 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "87", 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:18:06.457 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 8, 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 87, 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:18:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:18:06.459 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 3, 2024-09-21T15:18:06.460 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 3, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 3, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 3, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "new_default_pool17425", 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-09-21T15:16:34.640599+0000", 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 8193, 2024-09-21T15:18:06.461 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 8, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 8, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 8, 2024-09-21T15:18:06.462 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 8, 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 8, 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.463 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "54", 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 3, 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 54, 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-09-21T15:18:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-09-21T15:18:06.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "options": {}, 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-09-21T15:18:06.466 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd": {} 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 3, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 3, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 3, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 3, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.95147454738616943, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:18:06.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "6194913c-995a-4f8b-b2e4-1ae24c7fc1c6", 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 68, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:18:06.468 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.469 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.470 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.471 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3600997419 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6817/3600997419", 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/3600997419", 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6823/3600997419", 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6821/3600997419", 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "496a596a-f018-4010-bd93-6df3cfbef67f", 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:18:06.473 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 94, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1817390160 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6809/1817390160", 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6811/1817390160", 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6815/1817390160", 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6813/1817390160", 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:18:06.478 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "0131c1a7-80b4-448b-b7b2-89e6ba9ec584", 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:18:06.479 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 91, 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.480 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.482 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3371454484 2024-09-21T15:18:06.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/3371454484", 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/3371454484", 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/3371454484", 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/3371454484", 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:18:06.484 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "edacb78d-2a42-4dd8-a648-86031fd9ad18", 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0.7088470458984375, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 51, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6808", 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6809", 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.486 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6810", 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6811", 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6814", 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6815", 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.488 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6812", 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6813", 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1069930087 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6809/1069930087", 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6811/1069930087", 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6815/1069930087", 2024-09-21T15:18:06.489 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6813/1069930087", 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "bd5f5e45-2c13-432b-9aba-69c731ca2248", 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:18:06.490 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 12, 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 99, 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6816", 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.491 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6817", 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6818", 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6819", 2024-09-21T15:18:06.492 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6822", 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6823", 2024-09-21T15:18:06.493 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6820", 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.494 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6821", 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3606282773 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6817/3606282773", 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6819/3606282773", 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6823/3606282773", 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6821/3606282773", 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "52ceefef-20a8-41fb-ab77-053021d17105", 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 62, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-09-21T15:18:06.496 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6800", 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6801", 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.497 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6802", 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6803", 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.498 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6806", 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6807", 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.499 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6804", 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.142:6805", 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 3327577487 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.500 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.142:6801/3327577487", 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.142:6803/3327577487", 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.142:6807/3327577487", 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.142:6805/3327577487", 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:18:06.501 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.386986+0000", 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:18:06.502 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.358863+0000", 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.645118+0000", 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:11.318690+0000", 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:18:06.504 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.618213+0000", 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540138322906710015, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-09-21T15:15:10.843287+0000", 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.505 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.1", 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: 0 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.1", 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: 2, 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: 3 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.5", 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: 4, 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: 1 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "2.4", 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-09-21T15:18:06.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 4 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-09-21T15:18:06.508 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-09-21T15:18:06.509 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:18:06.510 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:18:06.510 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-09-21T15:18:06.510 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.510 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:06.512 INFO:tasks.thrashosds.thrasher:j is {'epoch': 101, 'fsid': 'fa0b2d62-c633-4668-b5db-f1f6328d1af9', 'created': '2024-09-21T15:15:04.255598+0000', 'modified': '2024-09-21T15:18:04.060642+0000', 'last_up_change': '2024-09-21T15:15:15.737314+0000', 'last_in_change': '2024-09-21T15:17:42.447038+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:15:15.995966+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:15:21.158896+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 87, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'new_default_pool17425', 'create_time': '2024-09-21T15:16:34.640599+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 54, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9514745473861694, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '6194913c-995a-4f8b-b2e4-1ae24c7fc1c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 68, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 3600997419}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6818', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 3600997419}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6822', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6823', 'nonce': 3600997419}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 3600997419}, {'type': 'v1', 'addr': '172.21.15.67:6821', 'nonce': 3600997419}]}, 'public_addr': '172.21.15.67:6817/3600997419', 'cluster_addr': '172.21.15.67:6819/3600997419', 'heartbeat_back_addr': '172.21.15.67:6823/3600997419', 'heartbeat_front_addr': '172.21.15.67:6821/3600997419', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '496a596a-f018-4010-bd93-6df3cfbef67f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6809', 'nonce': 1817390160}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1817390160}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6815', 'nonce': 1817390160}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6812', 'nonce': 1817390160}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 1817390160}]}, 'public_addr': '172.21.15.67:6809/1817390160', 'cluster_addr': '172.21.15.67:6811/1817390160', 'heartbeat_back_addr': '172.21.15.67:6815/1817390160', 'heartbeat_front_addr': '172.21.15.67:6813/1817390160', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '0131c1a7-80b4-448b-b7b2-89e6ba9ec584', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 3371454484}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 3371454484}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 3371454484}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 3371454484}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 3371454484}]}, 'public_addr': '172.21.15.67:6801/3371454484', 'cluster_addr': '172.21.15.67:6803/3371454484', 'heartbeat_back_addr': '172.21.15.67:6807/3371454484', 'heartbeat_front_addr': '172.21.15.67:6805/3371454484', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'edacb78d-2a42-4dd8-a648-86031fd9ad18', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7088470458984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 51, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6808', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6809', 'nonce': 1069930087}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6810', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6811', 'nonce': 1069930087}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6814', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6815', 'nonce': 1069930087}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6812', 'nonce': 1069930087}, {'type': 'v1', 'addr': '172.21.15.142:6813', 'nonce': 1069930087}]}, 'public_addr': '172.21.15.142:6809/1069930087', 'cluster_addr': '172.21.15.142:6811/1069930087', 'heartbeat_back_addr': '172.21.15.142:6815/1069930087', 'heartbeat_front_addr': '172.21.15.142:6813/1069930087', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd5f5e45-2c13-432b-9aba-69c731ca2248', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6816', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6817', 'nonce': 3606282773}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6818', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6819', 'nonce': 3606282773}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6822', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6823', 'nonce': 3606282773}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6820', 'nonce': 3606282773}, {'type': 'v1', 'addr': '172.21.15.142:6821', 'nonce': 3606282773}]}, 'public_addr': '172.21.15.142:6817/3606282773', 'cluster_addr': '172.21.15.142:6819/3606282773', 'heartbeat_back_addr': '172.21.15.142:6823/3606282773', 'heartbeat_front_addr': '172.21.15.142:6821/3606282773', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '52ceefef-20a8-41fb-ab77-053021d17105', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6800', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6801', 'nonce': 3327577487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6802', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6803', 'nonce': 3327577487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6806', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6807', 'nonce': 3327577487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6804', 'nonce': 3327577487}, {'type': 'v1', 'addr': '172.21.15.142:6805', 'nonce': 3327577487}]}, 'public_addr': '172.21.15.142:6801/3327577487', 'cluster_addr': '172.21.15.142:6803/3327577487', 'heartbeat_back_addr': '172.21.15.142:6807/3327577487', 'heartbeat_front_addr': '172.21.15.142:6805/3327577487', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.386986+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.358863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.645118+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:11.318690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.618213+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:15:10.843287+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.1', 'osds': [1, 0]}, {'pgid': '3.1', 'osds': [2, 3]}, {'pgid': '3.5', 'osds': [4, 1]}], 'pg_upmap_items': [{'pgid': '2.4', 'mappings': [{'from': 5, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:18:06.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:18:06.513 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:06.513 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:06.513 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:06.513 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:06.513 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:06.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:06.514 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:06.510+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:06.612 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:06.610+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:06.669 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:06.669 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:06.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:06.669 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 163.84768131199999, 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:18:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:18:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.285256810000007, 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.274520308999996, 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.679 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:18:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.683 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.230063474999994, 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:06.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:18:06.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.222286672999999, 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:18:06.692 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.183113001999999, 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:06.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:18:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.176380640999994, 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:06.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:18:06.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:18:06.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.142329149000005, 2024-09-21T15:18:06.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:18:06.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.132203165000007, 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:18:06.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:18:06.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.435439780999999, 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:18:06.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:18:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:06.714 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.298060477999996, 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:18:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:18:06.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.297236425999998, 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:06.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:18:06.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:18:06.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.412174878000002, 2024-09-21T15:18:06.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:18:06.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.411218806999997, 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:18:06.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.731 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 58.271543424000001, 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.977630148999999, 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:18:06.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:18:06.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.337691409000001, 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:06.740 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:06.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:06.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.746 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.415570589000001, 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:18:06.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.412844225000001, 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.750 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:18:06.753 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.410246698999998, 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:06.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:06.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:06.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:06.757 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:06.758 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:06.759 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:06.759 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:06.760+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:06.860 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:06.856+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:06.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:18:06.877 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:06.877 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:06.877 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:06.877 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:06.878 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-21T15:18:05.961343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147465123,"num_objects":77,"num_object_clones":19,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":13,"num_read":5722,"num_read_kb":155475,"num_write":936,"num_write_kb":273253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":167772443,"num_keys_recovered":45,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":890,"ondisk_log_size":890,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":464400,"kb_used_data":277704,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":565766640,"statfs":{"total":579820584960,"available":579345039360,"internally_reserved":0,"allocated":284368896,"data_stored":302041254,"data_compressed":25932,"data_compressed_allocated":368640,"data_compressed_original":9846784,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001741"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":244,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.037292+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:17:58.037292+0000","last_peered":"2024-09-21T15:17:58.037292+0000","last_clean":"2024-09-21T15:17:58.037292+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:17:58.037292+0000","last_undegraded":"2024-09-21T15:17:58.037292+0000","last_fullsized":"2024-09-21T15:17:58.037292+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'45","reported_seq":1140,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.037320+0000","last_change":"2024-09-21T15:16:55.672969+0000","last_active":"2024-09-21T15:17:58.037320+0000","last_peered":"2024-09-21T15:17:58.037320+0000","last_clean":"2024-09-21T15:17:58.037320+0000","last_became_active":"2024-09-21T15:16:55.272135+0000","last_became_peered":"2024-09-21T15:16:55.272135+0000","last_unstale":"2024-09-21T15:17:58.037320+0000","last_undegraded":"2024-09-21T15:17:58.037320+0000","last_fullsized":"2024-09-21T15:17:58.037320+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:58.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":20971558,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":960,"num_read_kb":31831,"num_write":42,"num_write_kb":32833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"75'6","reported_seq":100,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:02.062347+0000","last_change":"2024-09-21T15:17:44.260941+0000","last_active":"2024-09-21T15:18:02.062347+0000","last_peered":"2024-09-21T15:18:02.062347+0000","last_clean":"2024-09-21T15:18:02.062347+0000","last_became_active":"2024-09-21T15:17:43.480798+0000","last_became_peered":"2024-09-21T15:17:43.480798+0000","last_unstale":"2024-09-21T15:18:02.062347+0000","last_undegraded":"2024-09-21T15:18:02.062347+0000","last_fullsized":"2024-09-21T15:18:02.062347+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:07.595836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181755,"stat_sum":{"num_bytes":54,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":127,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"82'5","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:02.062219+0000","last_change":"2024-09-21T15:16:55.569580+0000","last_active":"2024-09-21T15:18:02.062219+0000","last_peered":"2024-09-21T15:18:02.062219+0000","last_clean":"2024-09-21T15:18:02.062219+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:02.062219+0000","last_undegraded":"2024-09-21T15:18:02.062219+0000","last_fullsized":"2024-09-21T15:18:02.062219+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:02.062181+0000","last_change":"2024-09-21T15:17:43.481624+0000","last_active":"2024-09-21T15:18:02.062181+0000","last_peered":"2024-09-21T15:18:02.062181+0000","last_clean":"2024-09-21T15:18:02.062181+0000","last_became_active":"2024-09-21T15:17:43.481181+0000","last_became_peered":"2024-09-21T15:17:43.481181+0000","last_unstale":"2024-09-21T15:18:02.062181+0000","last_undegraded":"2024-09-21T15:18:02.062181+0000","last_fullsized":"2024-09-21T15:18:02.062181+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:03:25.583980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"85'22","reported_seq":29,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.037421+0000","last_change":"2024-09-21T15:17:50.742335+0000","last_active":"2024-09-21T15:17:58.037421+0000","last_peered":"2024-09-21T15:17:58.037421+0000","last_clean":"2024-09-21T15:17:58.037421+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:17:58.037421+0000","last_undegraded":"2024-09-21T15:17:58.037421+0000","last_fullsized":"2024-09-21T15:17:58.037421+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":35,"num_read_kb":16384,"num_write":21,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":68,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.159637+0000","last_change":"2024-09-21T15:16:55.272512+0000","last_active":"2024-09-21T15:17:58.159637+0000","last_peered":"2024-09-21T15:17:58.159637+0000","last_clean":"2024-09-21T15:17:58.159637+0000","last_became_active":"2024-09-21T15:16:55.271874+0000","last_became_peered":"2024-09-21T15:16:55.271874+0000","last_unstale":"2024-09-21T15:17:58.159637+0000","last_undegraded":"2024-09-21T15:17:58.159637+0000","last_fullsized":"2024-09-21T15:17:58.159637+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:54:48.789474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046520300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"87'25","reported_seq":284,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.160470+0000","last_change":"2024-09-21T15:17:01.768225+0000","last_active":"2024-09-21T15:17:58.160470+0000","last_peered":"2024-09-21T15:17:58.160470+0000","last_clean":"2024-09-21T15:17:58.160470+0000","last_became_active":"2024-09-21T15:17:01.327838+0000","last_became_peered":"2024-09-21T15:17:01.327838+0000","last_unstale":"2024-09-21T15:17:58.160470+0000","last_undegraded":"2024-09-21T15:17:58.160470+0000","last_fullsized":"2024-09-21T15:17:58.160470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:08.047427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":12582969,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":132,"num_read_kb":8306,"num_write":24,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'1","reported_seq":73,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.160323+0000","last_change":"2024-09-21T15:16:55.676403+0000","last_active":"2024-09-21T15:17:58.160323+0000","last_peered":"2024-09-21T15:17:58.160323+0000","last_clean":"2024-09-21T15:17:58.160323+0000","last_became_active":"2024-09-21T15:16:55.274451+0000","last_became_peered":"2024-09-21T15:16:55.274451+0000","last_unstale":"2024-09-21T15:17:58.160323+0000","last_undegraded":"2024-09-21T15:17:58.160323+0000","last_fullsized":"2024-09-21T15:17:58.160323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:20.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046455,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"75'1","reported_seq":17,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-21T15:18:04.077483+0000","last_change":"2024-09-21T15:18:02.068119+0000","last_active":"2024-09-21T15:18:04.077483+0000","last_peered":"2024-09-21T15:18:04.077483+0000","last_clean":"2024-09-21T15:18:04.077483+0000","last_became_active":"2024-09-21T15:18:02.059405+0000","last_became_peered":"2024-09-21T15:18:02.059405+0000","last_unstale":"2024-09-21T15:18:04.077483+0000","last_undegraded":"2024-09-21T15:18:04.077483+0000","last_fullsized":"2024-09-21T15:18:04.077483+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:57:47.302331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":48,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"85'27","reported_seq":101,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:02.053408+0000","last_change":"2024-09-21T15:17:44.249295+0000","last_active":"2024-09-21T15:18:02.053408+0000","last_peered":"2024-09-21T15:18:02.053408+0000","last_clean":"2024-09-21T15:18:02.053408+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:02.053408+0000","last_undegraded":"2024-09-21T15:18:02.053408+0000","last_fullsized":"2024-09-21T15:18:02.053408+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":52,"num_read_kb":8203,"num_write":26,"num_write_kb":32770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"75'1","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:02.053672+0000","last_change":"2024-09-21T15:17:01.318184+0000","last_active":"2024-09-21T15:18:02.053672+0000","last_peered":"2024-09-21T15:18:02.053672+0000","last_clean":"2024-09-21T15:18:02.053672+0000","last_became_active":"2024-09-21T15:17:01.317913+0000","last_became_peered":"2024-09-21T15:17:01.317913+0000","last_unstale":"2024-09-21T15:18:02.053672+0000","last_undegraded":"2024-09-21T15:18:02.053672+0000","last_fullsized":"2024-09-21T15:18:02.053672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:47:53.254684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043867299999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"89'174","reported_seq":1335,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.037476+0000","last_change":"2024-09-21T15:17:24.302020+0000","last_active":"2024-09-21T15:17:58.037476+0000","last_peered":"2024-09-21T15:17:58.037476+0000","last_clean":"2024-09-21T15:17:58.037476+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:17:58.037476+0000","last_undegraded":"2024-09-21T15:17:58.037476+0000","last_fullsized":"2024-09-21T15:17:58.037476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":27287606,"num_objects":14,"num_object_clones":6,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":1248,"num_read_kb":38813,"num_write":192,"num_write_kb":52364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:02.053768+0000","last_change":"2024-09-21T15:17:43.480003+0000","last_active":"2024-09-21T15:18:02.053768+0000","last_peered":"2024-09-21T15:18:02.053768+0000","last_clean":"2024-09-21T15:18:02.053768+0000","last_became_active":"2024-09-21T15:17:43.479431+0000","last_became_peered":"2024-09-21T15:17:43.479431+0000","last_unstale":"2024-09-21T15:18:02.053768+0000","last_undegraded":"2024-09-21T15:18:02.053768+0000","last_fullsized":"2024-09-21T15:18:02.053768+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:19.528788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092366200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"89'244","reported_seq":1258,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:03.630735+0000","last_change":"2024-09-21T15:17:05.211552+0000","last_active":"2024-09-21T15:18:03.630735+0000","last_peered":"2024-09-21T15:18:03.630735+0000","last_clean":"2024-09-21T15:18:03.630735+0000","last_became_active":"2024-09-21T15:17:05.209676+0000","last_became_peered":"2024-09-21T15:17:05.209676+0000","last_unstale":"2024-09-21T15:18:03.630735+0000","last_undegraded":"2024-09-21T15:18:03.630735+0000","last_fullsized":"2024-09-21T15:18:03.630735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:02:52.206876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18923558,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":2396,"num_read_kb":34834,"num_write":141,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":20,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-09-21T15:18:02.053652+0000","last_change":"2024-09-21T15:17:43.476863+0000","last_active":"2024-09-21T15:18:02.053652+0000","last_peered":"2024-09-21T15:18:02.053652+0000","last_clean":"2024-09-21T15:18:02.053652+0000","last_became_active":"2024-09-21T15:17:43.476485+0000","last_became_peered":"2024-09-21T15:17:43.476485+0000","last_unstale":"2024-09-21T15:18:02.053652+0000","last_undegraded":"2024-09-21T15:18:02.053652+0000","last_fullsized":"2024-09-21T15:18:02.053652+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:37.900423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"89'222","reported_seq":483,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:17:58.037279+0000","last_change":"2024-09-21T15:17:24.303933+0000","last_active":"2024-09-21T15:17:58.037279+0000","last_peered":"2024-09-21T15:17:58.037279+0000","last_clean":"2024-09-21T15:17:58.037279+0000","last_became_active":"2024-09-21T15:17:24.303534+0000","last_became_peered":"2024-09-21T15:17:24.303534+0000","last_unstale":"2024-09-21T15:17:58.037279+0000","last_undegraded":"2024-09-21T15:17:58.037279+0000","last_fullsized":"2024-09-21T15:17:58.037279+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:39.035665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":20971596,"num_objects":11,"num_object_clones":4,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":4,"num_read":719,"num_read_kb":16966,"num_write":321,"num_write_kb":53462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":175,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146874685,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":13,"num_read":5568,"num_read_kb":155345,"num_write":771,"num_write_kb":270831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":167772268,"num_keys_recovered":29,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277250048,"data_stored":298156666,"data_compressed":14242,"data_compressed_allocated":294912,"data_compressed_original":8699904,"omap_allocated":0,"internal_metadata":0},"log_size":764,"ondisk_log_size":764,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40492,"kb_used_data":9376,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94331348,"statfs":{"total":96636764160,"available":96595300352,"internally_reserved":0,"allocated":9601024,"data_stored":9422884,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.54300000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.133,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.75600000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.032,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.82699999999999996,"15min":0.83699999999999997},"min":{"1min":0.53100000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2070000000000001,"5min":1.212,"15min":1.212},"last":0.85199999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65300000000000002,"15min":0.64500000000000002},"min":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":1.02},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84699999999999998,"15min":0.84999999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0880000000000001,"5min":1.208,"15min":1.208},"last":1.0860000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.82399999999999995,"15min":0.83599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0089999999999999,"5min":1.629,"15min":1.629},"last":1.292},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.1599999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56836,"kb_used_data":25720,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":26337280,"data_stored":25614834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.99099999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.83099999999999996},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96199999999999997,"5min":1.333,"15min":1.333},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.115,"5min":1.133,"15min":1.133},"last":0.85999999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.64200000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.84299999999999997,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.038,"5min":1.581,"15min":1.581},"last":0.90600000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.88600000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81299999999999994,"15min":0.81499999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.872,"5min":0.86599999999999999,"15min":0.86499999999999999},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.81399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105148,"kb_used_data":74032,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94266692,"statfs":{"total":96636764160,"available":96529092608,"internally_reserved":0,"allocated":75808768,"data_stored":86105204,"data_compressed":12966,"data_compressed_allocated":184320,"data_compressed_original":4923392,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.42899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.90300000000000002,"5min":1.226,"15min":1.226},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84699999999999998,"15min":0.84599999999999997},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.272,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.749},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86499999999999999,"15min":0.85999999999999999},"min":{"1min":0.64900000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.80100000000000005}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":1.2589999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.83499999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94768,"kb_used_data":63652,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94277072,"statfs":{"total":96636764160,"available":96539721728,"internally_reserved":0,"allocated":65179648,"data_stored":64456313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57799999999999996,"15min":0.59199999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.88300000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95499999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.83999999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91300000000000003,"5min":1.101,"15min":1.101},"last":0.628},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.90100000000000002,"15min":0.91500000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.085,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.90400000000000003,"15min":0.92200000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0289999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.75},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.91900000000000004,"15min":0.92900000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1839999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.77100000000000002}]}]},{"osd":4,"up_from":12,"seq":51539607591,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110284,"kb_used_data":79168,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94261556,"statfs":{"total":96636764160,"available":96523833344,"internally_reserved":0,"allocated":81068032,"data_stored":90751451,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.76100000000000001,"15min":0.77300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90200000000000002,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.76400000000000001,"15min":0.78100000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.331,"15min":1.331},"last":0.79000000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.82899999999999996,"15min":0.84699999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.86899999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.80500000000000005,"15min":0.81100000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.101,"5min":1.298,"15min":1.298},"last":0.86899999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.04,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78100000000000003,"15min":0.78700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93500000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.371}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82599999999999996,"15min":0.83699999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.048,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.76500000000000001}]}]},{"osd":2,"up_from":12,"seq":51539607589,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56872,"kb_used_data":25756,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94314968,"statfs":{"total":96636764160,"available":96578527232,"internally_reserved":0,"allocated":26374144,"data_stored":25690568,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.44},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.70099999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97399999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.48799999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.82999999999999996,"15min":0.83799999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.75},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.093,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.81499999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95999999999999996,"15min":0.96799999999999997},"min":{"1min":0.622,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.94799999999999995,"15min":0.95499999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":64032768,"data_stored":64012459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":74592256,"data_stored":85065896,"data_compressed":7121,"data_compressed_allocated":147456,"data_compressed_original":4349952,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25182208,"data_stored":25231452,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":79880192,"data_stored":90292373,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:18:06.880 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [2, 3, 1, 0] 2024-09-21T15:18:06.880 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.3', '2', '3', '1', '0'] 2024-09-21T15:18:06.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.3 2 3 1 0 2024-09-21T15:18:06.887 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:06.961 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:06.960+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:06.998 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:06.998 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:06.998 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:06.998 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:06.999 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:07.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:07.062 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:07.062+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:18:07.162 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:07.159+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:07.166 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.539941784000007, 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:18:07.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.423793846999999, 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:18:07.171 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.172 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.173 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.174 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:18:07.175 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.377173450000001, 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:07.176 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.177 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.178 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.179 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.326556826000001, 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:18:07.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:18:07.182 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.183 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.238203354000007, 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:18:07.186 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:18:07.187 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.749072483999996, 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.188 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.189 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:18:07.190 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:18:07.191 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.034191641999996, 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:18:07.192 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.193 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:18:07.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.195 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.196 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.945877385000003, 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:07.197 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.198 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.199 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.563905555999995, 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:18:07.201 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.202 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:18:07.205 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.678267106999996, 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:07.206 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.207 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:18:07.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.826531778000003, 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.676739917999999, 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029737039, 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487914+0000", 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489004+0000", 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010899900000000001 2024-09-21T15:18:07.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489031+0000", 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7645e-05 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489053+0000", 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489151+0000", 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7928000000000002e-05 2024-09-21T15:18:07.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489165+0000", 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3451999999999999e-05 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.517652+0000", 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028487375999999998 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.614389810999999, 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.221 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.352185939000002, 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:07.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:18:07.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:18:07.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.319135980999999, 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:18:07.225 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.230278705, 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:18:07.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.161025239000001, 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.233 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:18:07.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:18:07.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.117137670999998, 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:18:07.237 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.238 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 23.072256962000001, 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 22.979826386999999, 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:18:07.245 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.249 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:07.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:07.297 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:07.295+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:07.365 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:07.375 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:07.375 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:07.375 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:07.376 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:07.363+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:07.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:07.466 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:07.466+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:07.478 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:07.478 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:07.478 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:07.478 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:07.478 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:07.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:07.568 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:07.563+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:07.598 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:07.598 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:07.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:07.598 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.333051103000003, 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:18:07.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.602 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.258825887, 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:07.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.248426448000004, 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:07.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:18:07.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:18:07.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.186119442000006, 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:18:07.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:18:07.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.614 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 64.123909373999993, 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:07.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:18:07.618 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.426774137999999, 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.619 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:18:07.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.42632416, 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:18:07.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.426074294999999, 2024-09-21T15:18:07.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:18:07.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:18:07.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.425798790999998, 2024-09-21T15:18:07.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:07.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:18:07.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.425395825999999, 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:18:07.642 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:18:07.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.425189789999997, 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.648 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:18:07.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:18:07.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:18:07.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.424985786999997, 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 51.424794904999999, 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.659 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:18:07.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.663 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:07.665 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.287731579000003, 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:07.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.669 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:18:07.671 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.287496556999997, 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:18:07.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:18:07.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.287296134000002, 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:18:07.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:18:07.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:18:07.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.286892551999998, 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:18:07.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.286633395000003, 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:18:07.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:18:07.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:07.695 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.286438167999997, 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.698 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.699 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 50.092186404000003, 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:07.701 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:07.702 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.703 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.704 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:18:07.705 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:18:07.706 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.707 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.707 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.707 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:07.707 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:07.707 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:07.708 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:07.709 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:07.706+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:07.809 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:07.806+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:07.822 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:07.822 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:07.822 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:07.822 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:07.832 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:07.910 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:07.910+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:07.941 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:07.941 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:07.941 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:07.941 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:07.941 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:07.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:08.010 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:08.007+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:08.061 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:08.061 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.958089583000003, 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.062 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.063 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.064 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:18:08.065 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.958038540000004, 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:18:08.066 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.067 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.068 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.069 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:08.070 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.942418997000004, 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.071 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:18:08.072 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:18:08.073 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.074 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.942392244000004, 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.075 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.076 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:18:08.077 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.078 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.773344140999995, 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.079 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:08.080 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:18:08.081 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:18:08.082 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:18:08.083 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.084 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.758199284, 2024-09-21T15:18:08.085 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:18:08.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:18:08.088 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:18:08.089 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.729052468000006, 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:18:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:18:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.719725562999997, 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.098 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:18:08.099 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:18:08.100 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:18:08.101 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:08.102 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.692089410999998, 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.103 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:18:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:18:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:18:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:18:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.681011706000007, 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.646242158999996, 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:18:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:18:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.118 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.757718077000007, 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.119 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:18:08.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:18:08.122 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.123 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.747463010000004, 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.125 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.126 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.127 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:18:08.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.715416012999995, 2024-09-21T15:18:08.130 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:18:08.134 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.135 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.703204060000004, 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.136 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.137 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.138 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.141 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.669308102000002, 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.142 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.143 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.144 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.145 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.630263232999994, 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.147 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:08.148 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:18:08.149 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.151 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.769820105999997, 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.153 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.154 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.155 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:18:08.156 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:18:08.157 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:18:08.158 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.758242598999999, 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.159 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.160 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.161 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.162 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.163 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:08.164 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 70.723831830999998, 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:18:08.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:18:08.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.169 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:08.170 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:08.170 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:set 2.3 pg_upmap_items mapping to [2->3,1->0] 2024-09-21T15:18:08.172 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:08.170+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:08.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:08.272 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:08.267+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:08.329 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:08.330 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:08.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:08.373 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:08.371+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:08.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:18:08.449 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:08.449 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:08.449 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:08.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:08.474 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:08.471+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:08.565 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.108407807999995, 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:18:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:18:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.060523551000003, 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:18:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.012351705, 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:18:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:18:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:18:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 83.654182602999995, 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:18:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:18:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:18:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 83.488198732000001, 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 83.302693462999997, 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.324054631999999, 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:18:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.323269754999998, 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.322729631000001, 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:18:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:18:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:18:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.322527214999994, 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.322331009999999, 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.322136886999999, 2024-09-21T15:18:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:18:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:18:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.267689665999995, 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:18:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.267557218999997, 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.267405839999995, 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:18:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:18:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.267278469000004, 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:18:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.132478469000006, 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.995452384999993, 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:18:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:18:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:18:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.820155481, 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.769862955999997, 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:08.673 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:08.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:08.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:08.679 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:08.680 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:08.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:08.681 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:08.681 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:08.681 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:08.681 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:08.682 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:08.678+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:08.686 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:08.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:08.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:08.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:08.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:08.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:08.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:08.781 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:08.779+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:08.795 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:08.795 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:08.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:08.882 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:08.879+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:08.916 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:08.916 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:08.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:08.983 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:08.982+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:09.034 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.035 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:09.084 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:09.083+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:09.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:18:09.185 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:09.183+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:09.197 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.197 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:09.285 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:09.282+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:09.312 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.313 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:09.387 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:09.386+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:09.474 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.474 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:09.529 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:09.527+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:09.587 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:09.583+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:09.600 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.600 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.611 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:09.688 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:09.686+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:09.726 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.726 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:09.789 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:09.786+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:09.845 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.845 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:09.889 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:09.887+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:09.967 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:09.967 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:09.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:09.990 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:09.987+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:10.073 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:10.073 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:10.083 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:10.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:10.090+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:10.110 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:18:10.191 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:10.187+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:10.239 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:10.239 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:10.248 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:10.292 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:10.291+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:10.355 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:10.356 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:10.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:10.393 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:10.391+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:10.478 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:10.478 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:10.487 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:10.519 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd1 2024-09-21T15:18:10.519 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:18:10.519 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:18:10.519 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:18:10.519 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:18:10.519 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T15:18:10.532 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:10.531+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:10.590 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:10.591 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:10.595 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:10.591+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:10.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:10.621 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=20273 2024-09-21T15:18:10.622 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 20273 2024-09-21T15:18:10.622 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 20273 -C rbd-nbd 2024-09-21T15:18:10.631 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:18:10.631 INFO:tasks.workunit.client.0.smithi146.stdout: 20273 ? 00:00:00 rbd-nbd 2024-09-21T15:18:10.632 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-21T15:18:10.632 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:10.632 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:10.633 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:10.634 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:18:10.635 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:18:10.636 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:10.636 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-09-21T15:18:10.641 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-21T15:18:10.641 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:10.642 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:10.643 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:10.644 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:18:10.645 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:18:10.646 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:10.646 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-21T15:18:10.660 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-21T15:18:10.660 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:10.661 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:10.661 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:10.661 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:18:10.664 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:18:10.664 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:10.664 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-09-21T15:18:10.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-21T15:18:10.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:10.678 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:10.680 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:10.680 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:18:10.682 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:18:10.682 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:10.682 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-09-21T15:18:10.692 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-21T15:18:10.693 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:10.693 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:10.694 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:10.694 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:18:10.696 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:10.695+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:10.697 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:18:10.697 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:10.697 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-09-21T15:18:10.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:18:10.710 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-21T15:18:10.710 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:10.711 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:10.712 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:10.712 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:18:10.714 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:18:10.714 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:10.714 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-09-21T15:18:10.728 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:10.728 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:10.729 INFO:tasks.workunit.client.0.smithi146.stderr:+ tries=0 2024-09-21T15:18:10.730 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17425 2024-09-21T15:18:10.730 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T15:18:10.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:10.796 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:10.795+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:10.834 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:10.829+0000 7ff8a4c83700 -1 JournalMetadata: 0x559b018c1030 failed to locate client: INSPECT 2024-09-21T15:18:10.838 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:10.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ queue_length=4 2024-09-21T15:18:10.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 0 -lt 10 ']' 2024-09-21T15:18:10.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T15:18:10.842 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17425 --verbose 2024-09-21T15:18:10.849 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:10.849 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:10.849 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:10.849 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:10.849 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:10.856 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:10.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:10.874 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:10.896 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:10.897 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:10.894+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:10.920 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:10.937 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T15:18:10.937 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:10.937 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:10.937 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:10.937 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.652686+0000" 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.672054+0000" 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:10.938 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.704063+0000" 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:10.939 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:10.940 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:10.940 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:10.940 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.723212+0000" 2024-09-21T15:18:10.940 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:10.940 INFO:tasks.workunit.client.0.smithi146.stdout:Summary: 2024-09-21T15:18:10.940 INFO:tasks.workunit.client.0.smithi146.stdout: 4 entries inspected, 0 errors 2024-09-21T15:18:10.945 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:10.941+0000 7f120cf93700 -1 JournalMetadata: 0x556d67032530 failed to locate client: INSPECT 2024-09-21T15:18:10.951 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:10.952 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 1 2024-09-21T15:18:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.513253418999994, 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:10.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:10.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:10.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:10.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:18:10.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.977 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.465369162000002, 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:10.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:10.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:18:10.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:10.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:10.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.417197315999999, 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:10.983 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:10.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.059028213999994, 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:10.988 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:10.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:18:10.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:18:10.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.893044343, 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:18:10.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:18:10.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:18:10.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:18:10.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.707539073999996, 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:10.999 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:18:11.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:11.004 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.728900242999998, 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:18:11.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:18:11.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:18:11.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:18:11.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.728115365999997, 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.010 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:18:11.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.015 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.727575242, 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:11.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:18:11.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:18:11.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:18:11.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:18:11.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.727372826000007, 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:18:11.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:11.026 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.727176620999998, 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.726982497999998, 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.032 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:18:11.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:18:11.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:18:11.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.672535276999994, 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:18:11.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:18:11.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:18:11.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.672402829999996, 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:18:11.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:11.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:18:11.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:18:11.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:18:11.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.672251450999994, 2024-09-21T15:18:11.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:18:11.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:18:11.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.672124080000003, 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:18:11.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:18:11.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.537324080000005, 2024-09-21T15:18:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.064 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.400297996000006, 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:18:11.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.070 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.225001091999999, 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:18:11.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.174708566999996, 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.076 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.081 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:11.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:11.083 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:11.079+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:11.184 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:11.179+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:11.199 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:11.199 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:11.199 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:11.199 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:11.209 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:11.284 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:11.283+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:11.318 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:11.318 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:11.318 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:11.318 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:11.318 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:11.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:11.387 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:11.382+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:11.434 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 173.65140261400001, 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.435 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:18:11.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:18:11.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:11.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:18:11.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.974465391999999, 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.646446022999996, 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:18:11.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:18:11.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:18:11.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.595284579999998, 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:18:11.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.507238614000002, 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.457 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:11.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.462 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.926706835000005, 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:11.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:18:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:18:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:18:11.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.520462152999997, 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.957678383000001, 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:11.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:18:11.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:18:11.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:18:11.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.476 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.874969609999994, 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:11.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:18:11.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:18:11.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:18:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.832850336000007, 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:11.482 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:18:11.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:18:11.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:18:11.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:18:11.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.686359835000005, 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:18:11.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.590054413000004, 2024-09-21T15:18:11.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:18:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:18:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.039883940999999, 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:11.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.987045417000004, 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.501 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:18:11.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:18:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:18:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.109113739999998, 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.105552178000003, 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.511 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:18:11.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:18:11.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.101767004000003, 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.926732162, 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.038939782, 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:18:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:18:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:18:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.1 95/94 [PushOp(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head, version: 50'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head@50'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.481152849000001, 2024-09-21T15:18:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034730616999999998, 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952469+0000", 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953866+0000", 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013966040000000001 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953902+0000", 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6279000000000003e-05 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953930+0000", 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7988e-05 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954032+0000", 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102122 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954046+0000", 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3784000000000001e-05 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.987201+0000", 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033155234999999998 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.532 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:11.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:11.534 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:11.531+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:11.634 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:11.631+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:11.646 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:11.647 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:11.647 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:11.647 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:11.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:11.735 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:11.734+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:11.763 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:11.763 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:11.763 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:11.763 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:11.763 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:11.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:11.836 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:11.831+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:11.879 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:11.879 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:11.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 169.05787365200001, 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:18:11.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.495449149999999, 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.884 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:11.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:11.888 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.484712649000002, 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:18:11.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.440255815, 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:18:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:18:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.432479013000005, 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:18:11.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.393305342000005, 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.386572981, 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.352521488999997, 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:18:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.342395504999999, 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.918 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.645632121000006, 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:11.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:18:11.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.508252818000003, 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:11.923 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:18:11.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.507428766000004, 2024-09-21T15:18:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.931 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.622367218000001, 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:18:11.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:18:11.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:11.935 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.621411147000003, 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.939 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.481735764, 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.940 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:18:11.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:18:11.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.187822488999998, 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:18:11.944 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:18:11.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.547883749, 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.948 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:11.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:18:11.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.625762929, 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.623036565, 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:18:11.961 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.620439039000001, 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:11.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:18:11.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.965 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:11.966 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:11.966 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:11.967 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17425 2024-09-21T15:18:11.967 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T15:18:11.968 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:11.968 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:11.966+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:12.066 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:12.061+0000 7f8a1ade4700 -1 JournalMetadata: 0x556daf2ebb40 failed to locate client: INSPECT 2024-09-21T15:18:12.068 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:12.066+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:12.073 INFO:tasks.workunit.client.0.smithi146.stderr:+ queue_length=4 2024-09-21T15:18:12.074 INFO:tasks.workunit.client.0.smithi146.stderr:+ tries=1 2024-09-21T15:18:12.074 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -lt 10 ']' 2024-09-21T15:18:12.074 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T15:18:12.074 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17425 --verbose 2024-09-21T15:18:12.088 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:12.088 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:12.088 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:12.088 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:12.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:12.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.652686+0000" 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:12.168 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.672054+0000" 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.704063+0000" 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:12.169 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.723212+0000" 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:12.170 INFO:tasks.workunit.client.0.smithi146.stdout:Summary: 2024-09-21T15:18:12.171 INFO:tasks.workunit.client.0.smithi146.stdout: 4 entries inspected, 0 errors 2024-09-21T15:18:12.171 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:12.167+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:12.176 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:12.173+0000 7f5e2effd700 -1 JournalMetadata: 0x5636fb827000 failed to locate client: INSPECT 2024-09-21T15:18:12.183 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 1 2024-09-21T15:18:12.202 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:12.202 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:12.202 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:12.203 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:12.203 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:12.212 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:12.271 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:12.267+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.695533041999994, 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.321 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:18:12.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:12.325 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.579385105, 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:18:12.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:18:12.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.532764708000002, 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:18:12.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.333 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.482148084000002, 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:18:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.393794611999994, 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.904663741999997, 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.342 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:18:12.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:18:12.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.189782899999997, 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:18:12.346 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:12.350 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.101468643000004, 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:12.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:18:12.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:18:12.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.719496813999996, 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.356 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:18:12.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:18:12.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 77.833858364999998, 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:18:12.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:18:12.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.982123036000004, 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:18:12.363 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:18:12.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:18:12.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.832331176, 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029737039, 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487914+0000", 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489004+0000", 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010899900000000001 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489031+0000", 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7645e-05 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489053+0000", 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489151+0000", 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7928000000000002e-05 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489165+0000", 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3451999999999999e-05 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.517652+0000", 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028487375999999998 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.769981069, 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.507777196999999, 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.379 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.474727239, 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:12.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:18:12.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.385869963000001, 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:18:12.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.316616496999998, 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.391 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.272728928999999, 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.395 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.227848219999998, 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:12.396 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.397 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.398 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:12.399 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 28.135417645, 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:18:12.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:18:12.402 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.403 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:12.405 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:12.405 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:12.403+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:12.505 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:12.502+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:12.518 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:12.518 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:12.519 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:12.519 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:12.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:12.606 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:12.606+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:12.637 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:12.637 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:12.637 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:12.637 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:12.637 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:12.647 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:12.707 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:12.706+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:12.754 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.489211221000005, 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.758 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.414986005000003, 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.762 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.404586566000006, 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.766 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.342279559999994, 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:12.770 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 69.280069491999996, 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:18:12.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.582934256000001, 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.774 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:18:12.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:18:12.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.582484278000003, 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.785 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.582234413000002, 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:12.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:18:12.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:18:12.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:18:12.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.581958909000001, 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.791 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:18:12.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.581555944000002, 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.581349908, 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.802 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:12.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.807 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.581145905, 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.808 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:18:12.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 56.580955023000001, 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.814 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:12.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:18:12.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.443891696999998, 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:18:12.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:12.825 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.443656675, 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:12.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.443456251999997, 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.831 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:18:12.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:18:12.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:12.836 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.44305267, 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.838 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:18:12.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:18:12.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.841 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.442793512999998, 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:18:12.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:18:12.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.442598285999999, 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:18:12.847 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.852 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 55.248346521999999, 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:12.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:18:12.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:18:12.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:18:12.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:12.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:12.858 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:12.860 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:12.860 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:12.855+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:12.960 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:12.958+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:12.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:18:12.973 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:12.974 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:12.974 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:12.974 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:12.983 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:13.061 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:13.058+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:13.085 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:13.085 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:13.086 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:13.086 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:13.086 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:13.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:13.117 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:18:13.118 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:13.141 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:13.160 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:13.161 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:13.158+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:13.172 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:18:13.172 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:18:13.173 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:18:13.182 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:13.186 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17425 2024-09-21T15:18:13.186 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T15:18:13.203 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:13.207 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:13.207 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.102288474999995, 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:18:13.208 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.209 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.210 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.102237431999995, 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.212 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:13.213 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.214 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.215 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:13.216 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.086617888999996, 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:13.217 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:18:13.219 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.220 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.086591135999996, 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.221 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:13.222 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.223 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:18:13.224 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.225 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.917543033000001, 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.226 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.227 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:18:13.228 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:18:13.229 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:18:13.230 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.231 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.902398176000005, 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.232 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.233 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:18:13.234 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.235 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.236 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.237 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.873251359999998, 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.238 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.239 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.240 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.241 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:18:13.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.243 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.863924455000003, 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.244 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:13.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:18:13.246 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.247 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.249 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.836288303000003, 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.250 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.251 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.252 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.253 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.254 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.825210597999998, 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:18:13.255 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.256 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.257 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.258 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.259 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.260 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.790441051000002, 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.261 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:13.262 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.262 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.262 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.262 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.262 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:13.263 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:13.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.264 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.264 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:18:13.264 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:13.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:18:13.265 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.266 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.267 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.268 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.901916968999998, 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:13.269 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.270 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:18:13.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:18:13.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:18:13.273 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.891661901999996, 2024-09-21T15:18:13.274 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:18:13.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:18:13.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.859614905000001, 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.280 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:18:13.281 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.282 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.283 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.284 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.847402951999996, 2024-09-21T15:18:13.285 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:18:13.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:18:13.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:18:13.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.813506993999994, 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.291 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.296 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.774462124999999, 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.302 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.914018998000003, 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:18:13.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:18:13.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.902441491000005, 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 75.868030723000004, 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:18:13.314 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:13.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:18:13.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:13.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:13.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:13.320 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:13.321 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:13.321 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:13.259+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:13.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:13.285+0000 7f7aa90dd700 -1 JournalMetadata: 0x55e5663ca820 failed to locate client: INSPECT 2024-09-21T15:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:+ queue_length=4 2024-09-21T15:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:+ tries=2 2024-09-21T15:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 2 -lt 10 ']' 2024-09-21T15:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T15:18:13.322 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17425 --verbose 2024-09-21T15:18:13.370 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:13.370+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.652686+0000" 2024-09-21T15:18:13.394 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.672054+0000" 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:13.395 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.704063+0000" 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.723212+0000" 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout:Summary: 2024-09-21T15:18:13.396 INFO:tasks.workunit.client.0.smithi146.stdout: 4 entries inspected, 0 errors 2024-09-21T15:18:13.403 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:13.397+0000 7faa37fff700 -1 JournalMetadata: 0x55c4292b0f00 failed to locate client: INSPECT 2024-09-21T15:18:13.410 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 1 2024-09-21T15:18:13.453 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:13.453 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:13.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:13.464 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:13.463+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:13.515 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:13.518 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:18:13.531 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-21T15:18:13.109388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147465357,"num_objects":80,"num_object_clones":19,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":13,"num_read":6001,"num_read_kb":158282,"num_write":950,"num_write_kb":273260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":184615233,"num_keys_recovered":45,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":910,"ondisk_log_size":910,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":481212,"kb_used_data":294488,"kb_used_omap":9,"kb_used_meta":186614,"kb_avail":565749828,"statfs":{"total":579820584960,"available":579327823872,"internally_reserved":0,"allocated":301555712,"data_stored":319034072,"data_compressed":26027,"data_compressed_allocated":372736,"data_compressed_original":9912320,"omap_allocated":9561,"internal_metadata":191093415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":234,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":279,"num_read_kb":2807,"num_write":14,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16842790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.147521"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":250,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:18:09.099746+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:18:09.099746+0000","last_peered":"2024-09-21T15:18:09.099746+0000","last_clean":"2024-09-21T15:18:09.099746+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:09.099746+0000","last_undegraded":"2024-09-21T15:18:09.099746+0000","last_fullsized":"2024-09-21T15:18:09.099746+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"105'46","reported_seq":128,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-21T15:18:10.940706+0000","last_change":"2024-09-21T15:18:10.536161+0000","last_active":"2024-09-21T15:18:10.940706+0000","last_peered":"2024-09-21T15:18:10.940706+0000","last_clean":"2024-09-21T15:18:10.940706+0000","last_became_active":"2024-09-21T15:18:09.512431+0000","last_became_peered":"2024-09-21T15:18:09.512431+0000","last_unstale":"2024-09-21T15:18:10.940706+0000","last_undegraded":"2024-09-21T15:18:10.940706+0000","last_fullsized":"2024-09-21T15:18:10.940706+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:23:35.359631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001251383,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":1050,"num_read_kb":32953,"num_write":43,"num_write_kb":32834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":25231398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"75'6","reported_seq":103,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.092812+0000","last_change":"2024-09-21T15:17:44.260941+0000","last_active":"2024-09-21T15:18:08.092812+0000","last_peered":"2024-09-21T15:18:08.092812+0000","last_clean":"2024-09-21T15:18:08.092812+0000","last_became_active":"2024-09-21T15:17:43.480798+0000","last_became_peered":"2024-09-21T15:17:43.480798+0000","last_unstale":"2024-09-21T15:18:08.092812+0000","last_undegraded":"2024-09-21T15:18:08.092812+0000","last_fullsized":"2024-09-21T15:18:08.092812+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:07.595836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001181755,"stat_sum":{"num_bytes":54,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":127,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"82'5","reported_seq":75,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.092797+0000","last_change":"2024-09-21T15:16:55.569580+0000","last_active":"2024-09-21T15:18:08.092797+0000","last_peered":"2024-09-21T15:18:08.092797+0000","last_clean":"2024-09-21T15:18:08.092797+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:08.092797+0000","last_undegraded":"2024-09-21T15:18:08.092797+0000","last_fullsized":"2024-09-21T15:18:08.092797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388662,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":8,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.092871+0000","last_change":"2024-09-21T15:17:43.481624+0000","last_active":"2024-09-21T15:18:08.092871+0000","last_peered":"2024-09-21T15:18:08.092871+0000","last_clean":"2024-09-21T15:18:08.092871+0000","last_became_active":"2024-09-21T15:17:43.481181+0000","last_became_peered":"2024-09-21T15:17:43.481181+0000","last_unstale":"2024-09-21T15:18:08.092871+0000","last_undegraded":"2024-09-21T15:18:08.092871+0000","last_fullsized":"2024-09-21T15:18:08.092871+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:03:25.583980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"105'23","reported_seq":38,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-21T15:18:10.827641+0000","last_change":"2024-09-21T15:17:50.742335+0000","last_active":"2024-09-21T15:18:10.827641+0000","last_peered":"2024-09-21T15:18:10.827641+0000","last_clean":"2024-09-21T15:18:10.827641+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:18:10.827641+0000","last_undegraded":"2024-09-21T15:18:10.827641+0000","last_fullsized":"2024-09-21T15:18:10.827641+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582990,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":38,"num_read_kb":16386,"num_write":22,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":73,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:18:09.099483+0000","last_change":"2024-09-21T15:16:55.272512+0000","last_active":"2024-09-21T15:18:09.099483+0000","last_peered":"2024-09-21T15:18:09.099483+0000","last_clean":"2024-09-21T15:18:09.099483+0000","last_became_active":"2024-09-21T15:16:55.271874+0000","last_became_peered":"2024-09-21T15:16:55.271874+0000","last_unstale":"2024-09-21T15:18:09.099483+0000","last_undegraded":"2024-09-21T15:18:09.099483+0000","last_fullsized":"2024-09-21T15:18:09.099483+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:54:48.789474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046520300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"87'25","reported_seq":292,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-21T15:18:10.800101+0000","last_change":"2024-09-21T15:17:01.768225+0000","last_active":"2024-09-21T15:18:10.800101+0000","last_peered":"2024-09-21T15:18:10.800101+0000","last_clean":"2024-09-21T15:18:10.800101+0000","last_became_active":"2024-09-21T15:17:01.327838+0000","last_became_peered":"2024-09-21T15:17:01.327838+0000","last_unstale":"2024-09-21T15:18:10.800101+0000","last_undegraded":"2024-09-21T15:18:10.800101+0000","last_fullsized":"2024-09-21T15:18:10.800101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:06:08.047427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068153500000000004,"stat_sum":{"num_bytes":12582969,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":134,"num_read_kb":8308,"num_write":24,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12582950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"52'1","reported_seq":79,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:18:09.104183+0000","last_change":"2024-09-21T15:16:55.676403+0000","last_active":"2024-09-21T15:18:09.104183+0000","last_peered":"2024-09-21T15:18:09.104183+0000","last_clean":"2024-09-21T15:18:09.104183+0000","last_became_active":"2024-09-21T15:16:55.274451+0000","last_became_peered":"2024-09-21T15:16:55.274451+0000","last_unstale":"2024-09-21T15:18:09.104183+0000","last_undegraded":"2024-09-21T15:18:09.104183+0000","last_fullsized":"2024-09-21T15:18:09.104183+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:20.568480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001046455,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"75'1","reported_seq":20,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.089074+0000","last_change":"2024-09-21T15:18:02.068119+0000","last_active":"2024-09-21T15:18:08.089074+0000","last_peered":"2024-09-21T15:18:08.089074+0000","last_clean":"2024-09-21T15:18:08.089074+0000","last_became_active":"2024-09-21T15:18:02.059405+0000","last_became_peered":"2024-09-21T15:18:02.059405+0000","last_unstale":"2024-09-21T15:18:08.089074+0000","last_undegraded":"2024-09-21T15:18:08.089074+0000","last_fullsized":"2024-09-21T15:18:08.089074+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:57:47.302331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":48,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"85'27","reported_seq":104,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.088785+0000","last_change":"2024-09-21T15:17:44.249295+0000","last_active":"2024-09-21T15:18:08.088785+0000","last_peered":"2024-09-21T15:18:08.088785+0000","last_clean":"2024-09-21T15:18:08.088785+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:08.088785+0000","last_undegraded":"2024-09-21T15:18:08.088785+0000","last_fullsized":"2024-09-21T15:18:08.088785+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069194899999999997,"stat_sum":{"num_bytes":25165824,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":52,"num_read_kb":8203,"num_write":26,"num_write_kb":32770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"75'1","reported_seq":50,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.088815+0000","last_change":"2024-09-21T15:17:01.318184+0000","last_active":"2024-09-21T15:18:08.088815+0000","last_peered":"2024-09-21T15:18:08.088815+0000","last_clean":"2024-09-21T15:18:08.088815+0000","last_became_active":"2024-09-21T15:17:01.317913+0000","last_became_peered":"2024-09-21T15:17:01.317913+0000","last_unstale":"2024-09-21T15:18:08.088815+0000","last_undegraded":"2024-09-21T15:18:08.088815+0000","last_fullsized":"2024-09-21T15:18:08.088815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:47:53.254684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043867299999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"105'190","reported_seq":1514,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-21T15:18:10.947719+0000","last_change":"2024-09-21T15:17:24.302020+0000","last_active":"2024-09-21T15:18:10.947719+0000","last_peered":"2024-09-21T15:18:10.947719+0000","last_clean":"2024-09-21T15:18:10.947719+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:18:10.947719+0000","last_undegraded":"2024-09-21T15:18:10.947719+0000","last_fullsized":"2024-09-21T15:18:10.947719+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099023899999999996,"stat_sum":{"num_bytes":27287606,"num_objects":14,"num_object_clones":6,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":1426,"num_read_kb":40490,"num_write":202,"num_write_kb":52368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":64,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.088731+0000","last_change":"2024-09-21T15:17:43.480003+0000","last_active":"2024-09-21T15:18:08.088731+0000","last_peered":"2024-09-21T15:18:08.088731+0000","last_clean":"2024-09-21T15:18:08.088731+0000","last_became_active":"2024-09-21T15:17:43.479431+0000","last_became_peered":"2024-09-21T15:17:43.479431+0000","last_unstale":"2024-09-21T15:18:08.088731+0000","last_undegraded":"2024-09-21T15:18:08.088731+0000","last_fullsized":"2024-09-21T15:18:08.088731+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:34:19.528788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092366200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"89'244","reported_seq":1262,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.635029+0000","last_change":"2024-09-21T15:17:05.211552+0000","last_active":"2024-09-21T15:18:08.635029+0000","last_peered":"2024-09-21T15:18:08.635029+0000","last_clean":"2024-09-21T15:18:08.635029+0000","last_became_active":"2024-09-21T15:17:05.209676+0000","last_became_peered":"2024-09-21T15:17:05.209676+0000","last_unstale":"2024-09-21T15:18:08.635029+0000","last_undegraded":"2024-09-21T15:18:08.635029+0000","last_fullsized":"2024-09-21T15:18:08.635029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:02:52.206876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18923558,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":2396,"num_read_kb":34834,"num_write":141,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16777216,"num_keys_recovered":20,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":64,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:18:08.088873+0000","last_change":"2024-09-21T15:17:43.476863+0000","last_active":"2024-09-21T15:18:08.088873+0000","last_peered":"2024-09-21T15:18:08.088873+0000","last_clean":"2024-09-21T15:18:08.088873+0000","last_became_active":"2024-09-21T15:17:43.476485+0000","last_became_peered":"2024-09-21T15:17:43.476485+0000","last_unstale":"2024-09-21T15:18:08.088873+0000","last_undegraded":"2024-09-21T15:18:08.088873+0000","last_fullsized":"2024-09-21T15:18:08.088873+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:16:35.006816+0000","last_clean_scrub_stamp":"2024-09-21T15:16:35.006816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:37.900423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"105'224","reported_seq":494,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-21T15:18:10.827708+0000","last_change":"2024-09-21T15:17:24.303933+0000","last_active":"2024-09-21T15:18:10.827708+0000","last_peered":"2024-09-21T15:18:10.827708+0000","last_clean":"2024-09-21T15:18:10.827708+0000","last_became_active":"2024-09-21T15:17:24.303534+0000","last_became_peered":"2024-09-21T15:17:24.303534+0000","last_unstale":"2024-09-21T15:18:10.827708+0000","last_undegraded":"2024-09-21T15:18:10.827708+0000","last_fullsized":"2024-09-21T15:18:10.827708+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:50:39.035665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016005419999999999,"stat_sum":{"num_bytes":20971674,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":4,"num_read":725,"num_read_kb":16970,"num_write":323,"num_write_kb":53463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":175,"num_keys_recovered":16,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146874919,"num_objects":72,"num_object_clones":19,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":13,"num_read":5847,"num_read_kb":158152,"num_write":785,"num_write_kb":270838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184615058,"num_keys_recovered":29,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294055936,"data_stored":314991810,"data_compressed":14337,"data_compressed_allocated":299008,"data_compressed_original":8765440,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40584,"kb_used_data":9460,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94331256,"statfs":{"total":96636764160,"available":96595206144,"internally_reserved":0,"allocated":9687040,"data_stored":9458476,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.752,"15min":0.75600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.879}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.82599999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.133,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.99099999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.55600000000000005,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.032,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.82699999999999996,"15min":0.83699999999999997},"min":{"1min":0.53100000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2070000000000001,"5min":1.212,"15min":1.212},"last":1.0129999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.65300000000000002,"15min":0.64500000000000002},"min":{"1min":0.441,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.84699999999999998,"15min":0.84999999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0880000000000001,"5min":1.208,"15min":1.208},"last":0.85299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.82399999999999995,"15min":0.83599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0089999999999999,"5min":1.629,"15min":1.629},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.87,"15min":0.876},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.75}]}]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73312,"kb_used_data":42188,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94298528,"statfs":{"total":96636764160,"available":96561692672,"internally_reserved":0,"allocated":43200512,"data_stored":42488270,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.99099999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.83099999999999996},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.96199999999999997,"5min":1.333,"15min":1.333},"last":0.72799999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77100000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.115,"5min":1.133,"15min":1.133},"last":0.747}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.64200000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.84299999999999997,"15min":0.85099999999999998},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.038,"5min":1.581,"15min":1.581},"last":0.872}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.56100000000000005,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.81699999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.48699999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81299999999999994,"15min":0.81499999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.872,"5min":0.86599999999999999,"15min":0.86499999999999999},"min":{"1min":0.53300000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.67600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88832,"kb_used_data":57716,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94283008,"statfs":{"total":96636764160,"available":96545800192,"internally_reserved":0,"allocated":59101184,"data_stored":69284900,"data_compressed":12871,"data_compressed_allocated":180224,"data_compressed_original":4857856,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.438},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.42899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.36799999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.623,"15min":0.60999999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.81699999999999995,"15min":0.82899999999999996},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.90300000000000002,"5min":1.226,"15min":1.226},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.85499999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.84699999999999998,"15min":0.84599999999999997},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.272,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86499999999999999,"15min":0.85999999999999999},"min":{"1min":0.64900000000000002,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.75},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.71499999999999997,"5min":0.629,"15min":0.629},"max":{"1min":1.2589999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.626}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111256,"kb_used_data":80140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94260584,"statfs":{"total":96636764160,"available":96522838016,"internally_reserved":0,"allocated":82063360,"data_stored":81334929,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57799999999999996,"15min":0.59199999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68600000000000005,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73599999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.91600000000000004,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.86699999999999999,"15min":0.88300000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.95499999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.754}]},{"osd":4,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91300000000000003,"5min":1.101,"15min":1.101},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.90100000000000002,"15min":0.91500000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.085,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.81699999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.90400000000000003,"15min":0.92200000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0289999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.91900000000000004,"15min":0.92900000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1839999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.78700000000000003}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110312,"kb_used_data":79192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94261528,"statfs":{"total":96636764160,"available":96523804672,"internally_reserved":0,"allocated":81092608,"data_stored":90761639,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.76100000000000001,"15min":0.77300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90200000000000002,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.76400000000000001,"15min":0.78100000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88700000000000001,"5min":1.331,"15min":1.331},"last":0.76300000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.748},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90300000000000002,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.82899999999999996,"15min":0.84699999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.86899999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.82999999999999996}]},{"osd":2,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.80500000000000005,"15min":0.81100000000000005},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.101,"5min":1.298,"15min":1.298},"last":0.80500000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.04,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.78100000000000003,"15min":0.78700000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93500000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82599999999999996,"15min":0.83699999999999997},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0700000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.048,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.72299999999999998}]}]},{"osd":2,"up_from":12,"seq":51539607590,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56916,"kb_used_data":25792,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94314924,"statfs":{"total":96636764160,"available":96578482176,"internally_reserved":0,"allocated":26411008,"data_stored":25705858,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":1.028},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.74199999999999999,"15min":0.749},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.998}]},{"osd":1,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.67000000000000004,"15min":0.68300000000000005},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":0.70099999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.97399999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.92600000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.82999999999999996,"15min":0.83799999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Sep 21 15:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.65200000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.093,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.81599999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:17:22 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95999999999999996,"15min":0.96799999999999997},"min":{"1min":0.622,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":1.073},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.94799999999999995,"15min":0.95499999999999996},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.90500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":80830464,"data_stored":80855483,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":68215070,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25182208,"data_stored":25231452,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41959424,"data_stored":42008770,"data_compressed":95,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":79880192,"data_stored":90292373,"data_compressed":7026,"data_compressed_allocated":143360,"data_compressed_original":4284416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:18:13.566 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:13.566+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:13.569 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:13.569 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:13.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:13.667 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:13.663+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:13.681 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:13.681 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:13.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:13.768 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:13.766+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:13.803 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:13.803 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:13.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:13.868 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:13.867+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:13.964 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:13.965 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:13.969 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:13.966+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:13.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:14.070 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:14.067+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:14.084 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.084 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:14.171 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:14.170+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:14.200 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.200 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:14.272 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:14.270+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:14.312 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.312 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:14.373 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:14.371+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:14.413 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17425 2024-09-21T15:18:14.413 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T15:18:14.474 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:14.471+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:14.477 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.477 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:14.520 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:14.513+0000 7f700108d700 -1 JournalMetadata: 0x5650baff7eb0 failed to locate client: INSPECT 2024-09-21T15:18:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:+ queue_length=4 2024-09-21T15:18:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:+ tries=3 2024-09-21T15:18:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 3 -lt 10 ']' 2024-09-21T15:18:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T15:18:14.528 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd17425 --verbose 2024-09-21T15:18:14.574 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:14.571+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:14.606 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.606 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.617 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:14.623 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T15:18:14.623 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:14.623 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:14.623 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.652686+0000" 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 0, 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.672054+0000" 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:14.624 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.704063+0000" 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:14.625 INFO:tasks.workunit.client.0.smithi146.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout:{ 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout: "event_type": "AioDiscard", 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout: "offset": 4096, 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout: "length": 4096, 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout: "discard_granularity_bytes": 4096, 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout: "timestamp": "2024-09-21T15:18:10.723212+0000" 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout:} 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout:Summary: 2024-09-21T15:18:14.626 INFO:tasks.workunit.client.0.smithi146.stdout: 4 entries inspected, 0 errors 2024-09-21T15:18:14.633 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:14.629+0000 7ff2a1983700 -1 JournalMetadata: 0x55bba9fc6170 failed to locate client: INSPECT 2024-09-21T15:18:14.640 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 1 2024-09-21T15:18:14.676 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:14.671+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:14.718 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.718 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.728 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:14.778 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:14.775+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:14.835 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.835 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.844 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:14.879 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:14.878+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:14.954 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:14.954 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:14.964 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:15.012 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:15.011+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:15.071 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:15.071 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:15.080 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:15.075+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:15.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:15.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:18:15.180 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:15.179+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:15.185 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:15.185 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:15.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:15.281 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:15.279+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:15.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:18:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:15.344 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:15.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:15.382 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:15.382+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:15.461 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:15.464 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:15.464 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:15.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:15.479 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:15.483 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:15.479+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:15.500 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:15.520 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:15.541 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:15.563 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:15.565 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:15.583 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:15.582+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:15.588 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:15.588 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:15.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:15.588 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:15.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 94.482857078999999, 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:15.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 94.482806035999999, 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.593 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 94.467186493, 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:15.597 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:15.598 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.599 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.600 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 94.46715974, 2024-09-21T15:18:15.601 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.602 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.603 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.298111637000005, 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:18:15.606 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.607 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.608 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:18:15.609 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:18:15.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.609 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.609 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:18:15.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:18:15.611 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:18:15.612 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.282966779999995, 2024-09-21T15:18:15.613 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:18:15.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:18:15.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:18:15.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:18:15.618 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.253819964000002, 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.619 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.620 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.621 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.622 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.623 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:15.624 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.244493059000007, 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.625 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:18:15.626 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:18:15.627 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:18:15.628 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:18:15.629 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.216856906999993, 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.630 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.631 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.632 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.633 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:18:15.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.635 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.205779202000002, 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:15.636 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:18:15.637 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:18:15.638 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.639 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.640 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.171009655000006, 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.641 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.642 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:18:15.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:18:15.645 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.282485573000002, 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.647 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.648 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.649 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.650 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.651 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.272230506, 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.652 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.653 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.654 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.655 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.656 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.657 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.240183509000005, 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:18:15.659 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:18:15.660 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:18:15.661 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.662 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.227971556, 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.663 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.664 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:18:15.666 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.667 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.194075597999998, 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.669 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:18:15.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:18:15.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:18:15.673 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 90.155030729000003, 2024-09-21T15:18:15.674 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.675 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:18:15.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:18:15.679 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.294587602000007, 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:15.680 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.681 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:18:15.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:18:15.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.283010094999995, 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:18:15.686 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.688 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.689 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.691 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 78.248599326999994, 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.692 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:18:15.693 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:18:15.694 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:15.695 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:15.696 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:15.697 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:15.698 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd17425 2024-09-21T15:18:15.698 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T15:18:15.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:15.699 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:15.698+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:15.751 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:15.745+0000 7fcce9ffb700 -1 JournalMetadata: 0x55e7809561c0 failed to locate client: INSPECT 2024-09-21T15:18:15.759 INFO:tasks.workunit.client.0.smithi146.stderr:+ queue_length=0 2024-09-21T15:18:15.759 INFO:tasks.workunit.client.0.smithi146.stderr:+ tries=4 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 4 -lt 10 ']' 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 0 -gt 0 ']' 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 0 -eq 0 ']' 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd1 20273 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd1 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=20273 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:15.760 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:15.761 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:15.762 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:18:15.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:18:15.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:15.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:15.799 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:15.795+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:15.817 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:15.817 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:15.817 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:15.817 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:15.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:15.900 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:18:15.900 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^20273\b' 2024-09-21T15:18:15.900 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^20273\b' 2024-09-21T15:18:15.900 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:15.898+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:15.934 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:15.934 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:15.934 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:15.934 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:15.934 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:15.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:15.980 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:18:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *20273\b' 2024-09-21T15:18:15.981 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *20273\b' 2024-09-21T15:18:15.990 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:15.991 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:18:16.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:16.002+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:16.053 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.594233716000005, 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:18:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:18:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.546349458999998, 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.498177612999996, 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:18:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:18:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:18:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.140008511000005, 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:18:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.974024639999996, 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:18:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:18:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:18:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.788519371000007, 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.809880539999995, 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:18:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:18:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.809095662999994, 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:18:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:18:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:18:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.808555538999997, 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.808353123000003, 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:18:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:18:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:18:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.808156917999995, 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:18:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.807962794999995, 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:16.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.753515574000005, 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:18:16.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:18:16.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:18:16.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:18:16.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.753383127000006, 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:18:16.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.753231748000005, 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:18:16.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:18:16.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:18:16.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.753104377, 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:18:16.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:18:16.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.618304377000001, 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:18:16.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.481278293000003, 2024-09-21T15:18:16.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.305981388999996, 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:18:16.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.255688864000007, 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.158 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:16.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:18:16.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:18:16.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:18:16.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.164 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:16.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:18:16.166 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:16.162+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:16.266 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:16.262+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:16.323 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:16.324 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:16.324 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:16.324 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:16.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:16.382 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:16.382+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:16.441 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:16.441 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:16.441 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:16.441 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:16.441 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:16.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:16.467 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:16.467+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:16.492 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:18:16.492 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd feature disable rbd/testrbdnbd17425 journaling 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 178.78235879299999, 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 93.105421570999994, 2024-09-21T15:18:16.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:18:16.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:18:16.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.576 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.777402202000005, 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:18:16.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:18:16.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.726240759000007, 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:18:16.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:18:16.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.638194792999997, 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.588 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:18:16.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.057663013999999, 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.599 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.651418332000006, 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:18:16.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:18:16.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.088634561999996, 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.604 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:18:16.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:18:16.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.005925789000003, 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:18:16.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:18:16.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.963806515000002, 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:18:16.615 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:16.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:18:16.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.817316013999999, 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:18:16.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.626 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.721010591999999, 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:18:16.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:16.631 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.170840119999994, 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:18:16.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:18:16.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:18:16.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.118001595999999, 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.638 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:18:16.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:18:16.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:18:16.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.240069919, 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:18:16.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.236508356999998, 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:18:16.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:18:16.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.232723182999997, 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.652 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:18:16.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:18:16.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.057688341000002, 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:18:16.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:18:16.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:16.660 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.169895961000002, 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:16.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:18:16.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:18:16.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:18:16.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.1 95/94 [PushOp(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head, version: 50'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head@50'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.612109027999999, 2024-09-21T15:18:16.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034730616999999998, 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952469+0000", 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953866+0000", 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013966040000000001 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953902+0000", 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6279000000000003e-05 2024-09-21T15:18:16.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953930+0000", 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7988e-05 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954032+0000", 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102122 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954046+0000", 2024-09-21T15:18:16.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3784000000000001e-05 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.987201+0000", 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033155234999999998 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:16.670 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:16.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd config image rm rbd/testrbdnbd17425 rbd_discard_granularity_bytes 2024-09-21T15:18:16.672 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:16.667+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:16.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:16.730 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:18:16.730 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:18:16.731 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:18:16.733 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:18:16.733 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:18:16.736 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:18:16.736 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:18:16.736 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:18:16.773 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:16.771+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:16.792 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:16.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:16.792 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:16.792 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:16.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:16.874 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:16.871+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:16.910 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:16.910 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:16.910 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:16.910 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:16.911 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:16.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:16.976 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:16.971+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:17.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 174.20643434999999, 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:18:17.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:18:17.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.644009847999996, 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:18:17.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:18:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:18:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.633273346999999, 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.588816512999998, 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.042 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:18:17.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:18:17.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.581039711000003, 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.046 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:18:17.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:18:17.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.541866040000002, 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:18:17.050 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:18:17.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:18:17.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.535133678999998, 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:17.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.501082186999994, 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:18:17.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:18:17.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.061 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.490956202999996, 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.794192819000003, 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:18:17.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.656813516, 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:18:17.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:18:17.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:17.073 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.655989464000001, 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.770927916000005, 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:18:17.077 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:17.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:18:17.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:18:17.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.769971845000001, 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:17.081 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 68.630296462000004, 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:18:17.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.336383186999996, 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:17.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.696444446999998, 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.093 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:18:17.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:18:17.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:18:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.774323627000001, 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:17.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.771597262999997, 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:18:17.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.768999737000001, 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:17.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:17.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:17.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:18:17.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:18:17.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.109 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:17.109 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:17.109 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:17.110 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:17.111 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:17.110+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:17.198 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd1 2024-09-21T15:18:17.199 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:18:17.199 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:18:17.199 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:18:17.199 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:18:17.199 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T15:18:17.211 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:17.207+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:17.228 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:17.228 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:17.228 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:17.228 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:17.238 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:17.294 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=20712 2024-09-21T15:18:17.295 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 20712 2024-09-21T15:18:17.295 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 20712 -C rbd-nbd 2024-09-21T15:18:17.306 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:18:17.306 INFO:tasks.workunit.client.0.smithi146.stdout: 20712 ? 00:00:00 rbd-nbd 2024-09-21T15:18:17.306 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd feature enable rbd/testrbdnbd17425 journaling 2024-09-21T15:18:17.311 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:17.307+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:17.342 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:17.342 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:17.342 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:17.342 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:17.342 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:17.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:17.412 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:17.407+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:17.416 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-21T15:18:17.416 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:17.417 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:17.419 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:17.419 INFO:tasks.workunit.client.0.smithi146.stderr:++ which blkdiscard 2024-09-21T15:18:17.421 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T15:18:17.421 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:17.421 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-09-21T15:18:17.457 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd feature disable rbd/testrbdnbd17425 journaling 2024-09-21T15:18:17.463 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.838765273999996, 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:17.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.467 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.722617337000003, 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.675996940000005, 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.472 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:18:17.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.625380316000005, 2024-09-21T15:18:17.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:18:17.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.537026843999996, 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.483 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.484 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.047895973999999, 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:18:17.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.333015132, 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:17.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:18:17.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:18:17.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.244700875000007, 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.493 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:729 2.3 e59/48 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 59'21, mlcod=59'19)", 2024-09-21T15:18:17.497 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 91.862729045999998, 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029888057999999999, 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600749+0000", 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.600748+0000", 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601618+0000", 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087014600000000003 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601662+0000", 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4268999999999999e-05 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601674+0000", 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1914e-05 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601760+0000", 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.5654000000000004e-05 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.601792+0000", 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2493000000000002e-05 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630544+0000", 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028751912000000001 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.630637+0000", 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3200000000000002e-05 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:46 2.6 e62/48 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head v 62'96, mlcod=62'94)", 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 82.977090597, 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029864485, 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486388+0000", 2024-09-21T15:18:17.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.486386+0000", 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487200+0000", 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081406000000000002 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487226+0000", 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6429999999999999e-05 2024-09-21T15:18:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487244+0000", 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7672999999999998e-05 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487329+0000", 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4785999999999998e-05 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.487364+0000", 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5253000000000003e-05 2024-09-21T15:18:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516219+0000", 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028855261 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:54.516252+0000", 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2909e-05 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:18:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 80.125355268000007, 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:18:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.975563407999999, 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029737039, 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487915+0000", 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.487914+0000", 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489004+0000", 2024-09-21T15:18:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010899900000000001 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489031+0000", 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7645e-05 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489053+0000", 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1931999999999999e-05 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489151+0000", 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.7928000000000002e-05 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.489165+0000", 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3451999999999999e-05 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.517652+0000", 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028487375999999998 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.913213300999999, 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:18:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.517 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.651009428999998, 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.617959470999999, 2024-09-21T15:18:17.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:18:17.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:18:17.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.529102195, 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:17.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:18:17.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:18:17.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:17.528 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.459848729000001, 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.415961160999998, 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:18:17.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.535 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.371080452000001, 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:18:17.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 33.278649876999999, 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:17.539 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.542 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.543 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:17.543 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:17.543 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:17.544 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:17.544 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:17.542+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:17.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:18:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd1 20712 2024-09-21T15:18:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd1 2024-09-21T15:18:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=20712 2024-09-21T15:18:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:17.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:17.580 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:17.581 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:17.582 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:18:17.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:18:17.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:17.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:17.644 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:17.642+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:17.659 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:17.659 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:17.659 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:17.659 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:17.669 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:17.712 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:18:17.712 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^20712\b' 2024-09-21T15:18:17.712 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^20712\b' 2024-09-21T15:18:17.732 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:17.745 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:17.742+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:17.754 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:17.773 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:17.776 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:17.776 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:17.776 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:17.777 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:17.777 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:17.786 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:17.787 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *20712\b' 2024-09-21T15:18:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:18:17.788 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *20712\b' 2024-09-21T15:18:17.793 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:17.797 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:17.797 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:18:17.816 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:17.838 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:17.845 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:17.842+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:17.897 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:17.898 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:17.907 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:17.946 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:17.942+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:18.017 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.017 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:18.046 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:18.046+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:18.134 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.135 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.144 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:18.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:18.146+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:18.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:18:18.247 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:18.243+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:18.250 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.250 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:18.299 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:18:18.299 INFO:tasks.workunit.client.0.smithi146.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-09-21T15:18:18.330 INFO:tasks.workunit.client.0.smithi146.stderr:+ EXPECTED=2 2024-09-21T15:18:18.331 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:18:18.331 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:18:18.331 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:18:18.334 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:18:18.335 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:18:18.336 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:18:18.336 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:18:18.336 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd17425 2024-09-21T15:18:18.349 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:18.346+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:18.367 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.367 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:18.449 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:18.446+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:18.483 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.483 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:18.533 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:18:18.533 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:18:18.534 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-21T15:18:18.534 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:18:18.534 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:18:18.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-21T15:18:18.551 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:18.551+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:18.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd1 2024-09-21T15:18:18.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:18:18.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:18:18.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:18:18.579 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:18:18.579 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T15:18:18.601 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.601 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:18.643 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:18.644 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T15:18:18.644 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:18.652 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:18.651+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:18.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=20826 2024-09-21T15:18:18.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 20826 2024-09-21T15:18:18.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 20826 -C rbd-nbd 2024-09-21T15:18:18.687 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:18:18.688 INFO:tasks.workunit.client.0.smithi146.stdout: 20826 ? 00:00:00 rbd-nbd 2024-09-21T15:18:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:++ ps -p 20826 -T 2024-09-21T15:18:18.688 INFO:tasks.workunit.client.0.smithi146.stderr:++ grep -c io_context_pool 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ ACTUAL=2 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 2 -eq 2 ']' 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd1 20826 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd1 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=20826 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:18.701 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:18.702 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:18.703 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:18:18.704 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:18:18.704 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:18.704 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:18.714 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.714 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:18.753 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:18.751+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:18.815 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^20826\b' 2024-09-21T15:18:18.815 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:18:18.815 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^20826\b' 2024-09-21T15:18:18.835 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.835 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:18.854 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:18.851+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:18.891 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:18.891 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:18:18.892 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *20826\b' 2024-09-21T15:18:18.892 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *20826\b' 2024-09-21T15:18:18.901 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:18.901 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:18:18.955 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:18.954+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:18.956 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:18.956 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:18.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:19.056 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:19.055+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:19.076 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.076 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:18:19.156 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:19.154+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:19.197 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.198 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:19.257 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:19.255+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:19.311 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.312 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:19.358 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:19.358+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:19.403 INFO:tasks.workunit.client.0.smithi146.stderr:+ EXPECTED=7 2024-09-21T15:18:19.404 INFO:tasks.workunit.client.0.smithi146.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd17425 2024-09-21T15:18:19.404 INFO:tasks.workunit.client.0.smithi146.stderr:++ local cmd 2024-09-21T15:18:19.404 INFO:tasks.workunit.client.0.smithi146.stderr:+++ id -u 2024-09-21T15:18:19.406 INFO:tasks.workunit.client.0.smithi146.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T15:18:19.407 INFO:tasks.workunit.client.0.smithi146.stderr:+++ which rbd 2024-09-21T15:18:19.409 INFO:tasks.workunit.client.0.smithi146.stderr:++ cmd=/usr/bin/rbd 2024-09-21T15:18:19.410 INFO:tasks.workunit.client.0.smithi146.stderr:++ shift 2024-09-21T15:18:19.410 INFO:tasks.workunit.client.0.smithi146.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd17425 2024-09-21T15:18:19.441 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.442 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.452 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:19.458 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:19.455+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:19.559 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:19.558+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:19.562 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.562 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.573 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:19.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV=/dev/nbd1 2024-09-21T15:18:19.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ get_pid rbd 2024-09-21T15:18:19.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pool=rbd 2024-09-21T15:18:19.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns= 2024-09-21T15:18:19.640 INFO:tasks.workunit.client.0.smithi146.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T15:18:19.640 INFO:tasks.workunit.client.0.smithi146.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd17425'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T15:18:19.660 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:19.659+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:19.678 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.678 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.689 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:19.744 INFO:tasks.workunit.client.0.smithi146.stderr:+ PID=20893 2024-09-21T15:18:19.744 INFO:tasks.workunit.client.0.smithi146.stderr:+ test -n 20893 2024-09-21T15:18:19.744 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -p 20893 -C rbd-nbd 2024-09-21T15:18:19.752 INFO:tasks.workunit.client.0.smithi146.stdout: PID TTY TIME CMD 2024-09-21T15:18:19.752 INFO:tasks.workunit.client.0.smithi146.stdout: 20893 ? 00:00:00 rbd-nbd 2024-09-21T15:18:19.753 INFO:tasks.workunit.client.0.smithi146.stderr:++ ps -p 20893 -T 2024-09-21T15:18:19.753 INFO:tasks.workunit.client.0.smithi146.stderr:++ grep -c io_context_pool 2024-09-21T15:18:19.759 INFO:tasks.workunit.client.0.smithi146.stderr:+ ACTUAL=7 2024-09-21T15:18:19.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 7 -eq 7 ']' 2024-09-21T15:18:19.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ unmap_device /dev/nbd1 20893 2024-09-21T15:18:19.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ local args=/dev/nbd1 2024-09-21T15:18:19.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ local pid=20893 2024-09-21T15:18:19.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:19.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ local cmd 2024-09-21T15:18:19.761 INFO:tasks.workunit.client.0.smithi146.stderr:++ id -u 2024-09-21T15:18:19.761 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:19.759+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:19.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T15:18:19.762 INFO:tasks.workunit.client.0.smithi146.stderr:++ which rbd 2024-09-21T15:18:19.763 INFO:tasks.workunit.client.0.smithi146.stderr:+ cmd=/usr/bin/rbd 2024-09-21T15:18:19.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ shift 2024-09-21T15:18:19.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T15:18:19.843 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.843 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.853 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:19.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:18:19.876 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd device --device-type nbd list 2024-09-21T15:18:19.876 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^20893\b' 2024-09-21T15:18:19.876 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^20893\b' 2024-09-21T15:18:19.898 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:19.898+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:19.951 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:19.952 INFO:tasks.workunit.client.0.smithi146.stderr:+ expect_false grep '^ *20893\b' 2024-09-21T15:18:19.952 INFO:tasks.workunit.client.0.smithi146.stderr:+ ps -C rbd-nbd 2024-09-21T15:18:19.952 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep '^ *20893\b' 2024-09-21T15:18:19.963 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 0 2024-09-21T15:18:19.963 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:19.962+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:19.963 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:18:19.964 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:19.964 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:19.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:20.007 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:20.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:20.053 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:20.064 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:20.063+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:20.074 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:20.081 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:20.081 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:20.094 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:20.097 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:20.120 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:20.165 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:20.163+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:20.247 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:20.247 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:20.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:20.266 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:20.266+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:20.360 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:20.360 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:20.360 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:20.367 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:20.366+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:20.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:20.464 INFO:tasks.workunit.client.0.smithi146.stderr:+ DEV= 2024-09-21T15:18:20.465 INFO:tasks.workunit.client.0.smithi146.stdout:OK 2024-09-21T15:18:20.465 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo OK 2024-09-21T15:18:20.466 INFO:tasks.workunit.client.0.smithi146.stderr:+ cleanup 2024-09-21T15:18:20.466 INFO:tasks.workunit.client.0.smithi146.stderr:+ local ns s 2024-09-21T15:18:20.466 INFO:tasks.workunit.client.0.smithi146.stderr:+ set +e 2024-09-21T15:18:20.466 INFO:tasks.workunit.client.0.smithi146.stderr:+ mount 2024-09-21T15:18:20.466 INFO:tasks.workunit.client.0.smithi146.stderr:+ fgrep /tmp/tmp.flUinkcHzA/mnt 2024-09-21T15:18:20.467 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:20.466+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:20.477 INFO:tasks.workunit.client.0.smithi146.stderr:+ rm -Rf /tmp/tmp.flUinkcHzA 2024-09-21T15:18:20.482 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 99.377646494999993, 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:20.483 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:20.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 99.377595451999994, 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:18:20.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 99.361975908999995, 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.492 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:20.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.496 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 99.361949155999994, 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:18:20.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:18:20.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.192901053, 2024-09-21T15:18:20.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:18:20.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:18:20.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.506 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.507 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.177756196000004, 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:18:20.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:18:20.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.148609379999996, 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:18:20.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.517 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.519 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.139282475000002, 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:18:20.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.111646323000002, 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.525 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:18:20.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.100568617999997, 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.531 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:18:20.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.534 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.536 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 96.065799071000001, 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:18:20.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:18:20.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 95.177274988999997, 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:18:20.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:18:20.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:18:20.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 95.167019921999994, 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:18:20.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.554 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 95.134972925, 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.556 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 95.122760971999995, 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.560 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:18:20.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:18:20.563 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.564 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.565 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 95.088865014000007, 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.566 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:20.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:18:20.568 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.569 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.570 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:20.571 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 95.049820144999998, 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.572 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:18:20.573 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.574 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.575 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.576 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.189377018000002, 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.577 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:20.578 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:18:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.177799511000003, 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:18:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:18:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 83.143388743000003, 2024-09-21T15:18:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:18:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:20.595 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:20.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -n '' ']' 2024-09-21T15:18:20.597 INFO:tasks.workunit.client.0.smithi146.stderr:+ for ns in '' ${NS} 2024-09-21T15:18:20.597 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace '' status testrbdnbd17425 2024-09-21T15:18:20.597 INFO:tasks.workunit.client.0.smithi146.stdout:Watchers: none 2024-09-21T15:18:20.597 INFO:tasks.workunit.client.0.smithi146.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-21T15:18:20.598 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:18:20.598 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:20.595+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:20.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:20.697 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:20.694+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:20.718 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:20.718 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:20.718 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:20.718 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:20.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:20.798 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:20.798+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:20.839 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:20.840 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:20.840 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:20.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:20.899 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:20.898+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:21.000 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:20.995+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.549373544999995, 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.009 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:18:21.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.501489288000002, 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.015 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:18:21.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.453317441999999, 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:18:21.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:18:21.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:18:21.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.095148339999994, 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:18:21.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:18:21.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.929164469, 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.033 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:21.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:18:21.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:18:21.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.743659199999996, 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.039 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:18:21.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.765020368999998, 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.045 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:21.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.764235491999997, 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.051 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:18:21.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:18:21.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:18:21.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.763695368, 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:18:21.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:18:21.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.763492952000007, 2024-09-21T15:18:21.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:18:21.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:18:21.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.763296746999998, 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:18:21.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:18:21.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:18:21.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:18:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.763102623999998, 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:18:21.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:18:21.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:18:21.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.708655402999995, 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.708522955999996, 2024-09-21T15:18:21.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.708371576999994, 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:18:21.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:18:21.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:18:21.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:18:21.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.708244206000003, 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.094 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.573444206000005, 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.100 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:18:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:18:21.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:18:21.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.436418122000006, 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.105 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:18:21.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:21.110 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.261121218, 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:18:21.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:18:21.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:21.115 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.210828692999996, 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:18:21.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.120 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:21.122 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace '' status testrbdnbd17425 2024-09-21T15:18:21.122 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep 'Watchers: none' 2024-09-21T15:18:21.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:21.123 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:21.119+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:21.145 INFO:tasks.workunit.client.0.smithi146.stdout:Watchers: none 2024-09-21T15:18:21.145 INFO:tasks.workunit.client.0.smithi146.stderr:+ break 2024-09-21T15:18:21.146 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd17425 2024-09-21T15:18:21.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:18:21.222 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:21.222+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:21.241 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:21.241 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:21.241 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:21.241 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:21.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:21.323 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:21.319+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:21.364 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:21.364 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:21.364 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:21.364 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:21.364 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:21.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:21.424 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:21.423+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:21.486 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 183.70352225799999, 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:21.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:18:21.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 98.026585036, 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.494 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:18:21.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.698565666999997, 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:18:21.498 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:18:21.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:18:21.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.647404223999999, 2024-09-21T15:18:21.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:18:21.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:18:21.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:18:21.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.559358258000003, 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.510 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.978826479000006, 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:18:21.515 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.572581796999998, 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:18:21.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:18:21.523 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.009798027000002, 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.524 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:18:21.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:18:21.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:727 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.927089253999995, 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033648714000000003, 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.530 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 727 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558654+0000", 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.558653+0000", 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559979+0000", 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013263769999999999 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.559995+0000", 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5634e-05 2024-09-21T15:18:21.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560003+0000", 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560045+0000", 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2487000000000002e-05 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560121+0000", 2024-09-21T15:18:21.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5585000000000002e-05 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.560220+0000", 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9648000000000005e-05 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.591388+0000", 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031167758 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592178+0000", 2024-09-21T15:18:21.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00079021600000000005 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592253+0000", 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5131999999999998e-05 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.592303+0000", 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.935e-05 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.884969979999994, 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:18:21.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:18:21.541 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.738479479000006, 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.545 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.642174057000005, 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:18:21.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:21.550 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.092003585, 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:21.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:18:21.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.556 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.039165061000006, 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:18:21.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.161233383999999, 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:18:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:18:21.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:18:21.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:201 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'118, mlcod=76'112)", 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.157671821999998, 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033873309999999997, 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328071+0000", 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328070+0000", 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328126+0000", 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6023000000000002e-05 2024-09-21T15:18:21.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328140+0000", 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4513e-05 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.328148+0000", 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0310000000000001e-06 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335934+0000", 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077867520000000001 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.335960+0000", 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6137999999999999e-05 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361914+0000", 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025953673 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361945+0000", 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0691000000000001e-05 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.153886647999997, 2024-09-21T15:18:21.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:21.575 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.978851806000002, 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:384 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:21.579 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.091059426000001, 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035242982999999999, 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 384 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394684+0000", 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.394682+0000", 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396054+0000", 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371511 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396084+0000", 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0273e-05 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396096+0000", 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2591e-05 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396141+0000", 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4058999999999999e-05 2024-09-21T15:18:21.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396193+0000", 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2296000000000002e-05 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.396272+0000", 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8856000000000005e-05 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:21.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.419425+0000", 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023153232999999999 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429817+0000", 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010392340999999999 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429874+0000", 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6425999999999998e-05 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.429927+0000", 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3159999999999999e-05 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.1 95/94 [PushOp(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head, version: 50'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:9f464168:ns::rbd_data.137aae78387e.0000000000000008:head@50'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.533272492999998, 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034730616999999998, 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952471+0000", 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.952469+0000", 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953866+0000", 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013966040000000001 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953902+0000", 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6279000000000003e-05 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.953930+0000", 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7988e-05 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954032+0000", 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102122 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.954046+0000", 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3784000000000001e-05 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:49.987201+0000", 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033155234999999998 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.589 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:21.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:21.591 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:21.587+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:21.691 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:21.691+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:21.705 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:21.705 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:21.705 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:21.705 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:21.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:21.793 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:21.791+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:21.822 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:21.823 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:21.823 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:21.823 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:21.823 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:21.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:21.893 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:21.890+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:21.942 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:21.942 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:21.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 179.11971564800001, 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:18:21.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.557291145999997, 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.947 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:18:21.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.546554645000001, 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:18:21.952 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:18:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:18:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.502097810999999, 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:18:21.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:18:21.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.494321009000004, 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:18:21.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:18:21.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:21.965 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.455147338000003, 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:21.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.969 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.448414976999999, 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:18:21.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:18:21.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.414363484999996, 2024-09-21T15:18:21.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:18:21.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.978 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.404237500999997, 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:18:21.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.982 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.707474117000004, 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:21.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.570094814000001, 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.987 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.569270762000002, 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:21.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:18:21.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:21.994 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.684209214000006, 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:18:21.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:18:21.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:21.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:3 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.683253143000002, 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045016199999999999, 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:21.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258824+0000", 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258823+0000", 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258831+0000", 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1340000000000004e-06 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258851+0000", 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9717000000000001e-05 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258860+0000", 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3819999999999993e-06 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258932+0000", 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1978999999999996e-05 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259064+0000", 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013192600000000001 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.259275+0000", 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021022400000000001 2024-09-21T15:18:22.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 73.543577760000005, 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:18:22.003 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.249664484999997, 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:18:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:22.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:22.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.609725744999999, 2024-09-21T15:18:22.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:18:22.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:18:22.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:18:22.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.687604925000002, 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:18:22.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.684878560999998, 2024-09-21T15:18:22.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:22.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:22.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 37.682281035000003, 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:22.025 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:18:22.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:18:22.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:22.028 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:22.030 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:22.030 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:22.026+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:22.131 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:22.130+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:22.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:18:22.141 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:22.141 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:22.141 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:22.141 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:22.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-09-21T15:18:22.232 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:22.231+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:22.256 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:22.256 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:22.266 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:22.288 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:22.308 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:22.332 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:22.330+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:22.350 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:22.368 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:22.368 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:22.372 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:22.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:22.395 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:22.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:22.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:22.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:22.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:22.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:22.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:22.454 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:22.454+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:22.482 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:22.482 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:22.491 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:22.534 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:22.534+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:22.599 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:22.599 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:22.609 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:22.635 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:22.634+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:22.712 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:22.713 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:22.722 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:22.735 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:22.734+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:22.833 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:22.833 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:22.837 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:22.834+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:22.843 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:22.938 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:22.935+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:22.953 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:22.953 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:22.963 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:23.039 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:23.038+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:23.072 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.072 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:23.139 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:23.135+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:23.207 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.207 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.218 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:23.240 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:23.238+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:23.318 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.318 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:23.342 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:23.339+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:23.436 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.436 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.443 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:23.439+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:23.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:23.544 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:23.542+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:23.554 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.555 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:23.645 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:23.642+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:23.654 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:18:23.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:18:23.654 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T15:18:23.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-09-21T15:18:23.673 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.673 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:23.746 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:23.743+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:23.844 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.844 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.846 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:23.843+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:23.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:23.948 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:23.946+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:23.955 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:23.955 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:23.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2024-09-21T15:18:23.951+0000 7fac8205d700 -1 mgr.server reply reply (3) No such process no change 2024-09-21T15:18:23.956 INFO:teuthology.orchestra.run.smithi067.stdout:moved 2 / 34 (5.88235%) 2024-09-21T15:18:23.956 INFO:teuthology.orchestra.run.smithi067.stdout:avg 5.66667 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 1.24722 -> 1.10554 (expected baseline 2.17307) 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.5 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.4 with 8 -> 7 pgs (1.41176 -> 1.23529 * mean) 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:oload 110 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 1 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0008 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0009 2024-09-21T15:18:23.957 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 weight 1.0000 -> 0.6964 2024-09-21T15:18:23.958 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 weight 1.0000 -> 0.7023 2024-09-21T15:18:23.958 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 weight 1.0000 -> 0.8722 2024-09-21T15:18:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:18:23.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:23.972 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T15:18:23.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-21T15:18:24.049 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:24.046+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:24.103 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:24.103 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:24.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:24.150 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:24.147+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:24.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:18:24.236 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:24.236 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:24.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:24.249 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:24.246+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:24.350 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:24.347+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:24.359 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:24.359 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:24.369 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:24.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:18:24.454 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:24.454+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:24.477 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:24.478 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:24.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:24.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:24.551 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:24.550+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:24.570 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:24.589 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:24.593 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:24.593 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:24.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:24.613 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:24.634 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:24.652 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:24.651+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:24.657 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:24.712 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:24.712 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:24.712 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:24.712 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:24.722 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:24.752 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:24.750+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:24.825 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:24.825 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:24.825 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:24.825 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:24.825 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:24.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:24.853 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:24.851+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:24.944 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:24.944 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:24.944 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:24.944 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:24.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.944 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.678660773999994, 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.945 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.946 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.948 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head, version: 63'124, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:68fde513:::rbd_data.13743413a48b.000000000000000d:head@63'124, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.604435558000006, 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038651392999999999, 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.949 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337097+0000", 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.337096+0000", 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340551+0000", 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0034549569999999998 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.950 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340581+0000", 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0083999999999999e-05 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.340602+0000", 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0965e-05 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344285+0000", 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0036827299999999999 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.951 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.344308+0000", 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2657e-05 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375748+0000", 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031440834000000001 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:24.952 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.594036118999995, 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.953 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:18:24.954 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.955 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:24.956 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.531729112999997, 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:24.957 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:18:24.958 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.959 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:24.960 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 81.469519044999998, 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:18:24.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:18:24.963 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.772383809000004, 2024-09-21T15:18:24.964 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.965 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:18:24.966 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.967 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.968 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.969 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.771933830999998, 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.970 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.971 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.972 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.973 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:18:24.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.771683965999998, 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:24.976 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.977 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.978 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:18:24.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.981 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.771408461999997, 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.982 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:18:24.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.771005497000004, 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.987 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.988 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.989 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.990 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.991 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.992 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.770799460999996, 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.993 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:24.994 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.995 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.996 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.997 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.998 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.770595458000003, 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:24.999 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.000 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.001 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:18:25.002 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.003 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 68.770404576000004, 2024-09-21T15:18:25.004 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.005 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.006 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:18:25.007 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.008 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.009 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.633341250000001, 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.010 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:25.011 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.012 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:18:25.013 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:18:25.014 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.015 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.633106228000003, 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.016 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:25.017 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.018 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.019 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.020 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:18:25.021 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.632905805000007, 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.022 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.023 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:18:25.024 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.025 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.026 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:18:25.027 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.632502223000003, 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.028 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.029 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:18:25.030 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.031 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.032 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.033 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.632243066000001, 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.034 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.035 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.036 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:18:25.038 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.632047838999995, 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.040 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:25.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.042 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.044 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:25.045 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 67.437796074999994, 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:25.046 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:18:25.047 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:18:25.048 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:18:25.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:18:25.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.051 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.051 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.051 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.051 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.051 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:25.052 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:25.053 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:25.050+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:25.153 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:25.151+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:25.165 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:25.165 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:25.165 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:25.165 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:25.175 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:25.213 INFO:teuthology.orchestra.run.smithi067.stdout:moved 2 / 34 (5.88235%) 2024-09-21T15:18:25.213 INFO:teuthology.orchestra.run.smithi067.stdout:avg 5.66667 2024-09-21T15:18:25.213 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 1.24722 -> 1.10554 (expected baseline 2.17307) 2024-09-21T15:18:25.213 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.5 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-21T15:18:25.213 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.4 with 8 -> 7 pgs (1.41176 -> 1.23529 * mean) 2024-09-21T15:18:25.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:25.213 INFO:teuthology.orchestra.run.smithi067.stdout:oload 110 2024-09-21T15:18:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 3 2024-09-21T15:18:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:18:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0008 2024-09-21T15:18:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0009 2024-09-21T15:18:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 weight 1.0000 -> 0.6964 2024-09-21T15:18:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 weight 1.0000 -> 0.7023 2024-09-21T15:18:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 weight 1.0000 -> 0.8722 2024-09-21T15:18:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-09-21T15:18:25.231 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T15:18:25.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-09-21T15:18:25.254 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:25.251+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:25.287 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:25.288 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:25.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:25.355 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:25.351+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.348527877, 2024-09-21T15:18:25.454 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:18:25.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.348476834, 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.459 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:18:25.462 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.332857291, 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.463 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.464 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.465 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.466 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.332830538, 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:25.467 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:18:25.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:18:25.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.163782435, 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:18:25.471 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.472 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.473 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.474 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:25.477 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.14863757800001, 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:18:25.479 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:18:25.480 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:18:25.481 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.482 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.119490762, 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.483 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:18:25.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.486 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.487 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.110163857, 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:18:25.490 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.491 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.492 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.493 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.494 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.082527705, 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:25.495 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:18:25.496 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.497 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.498 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.499 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.07145, 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.500 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:25.501 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.502 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.503 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:18:25.504 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 101.036680453, 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.506 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.508 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.509 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.510 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 100.148156371, 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.511 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:25.512 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.513 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.514 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:18:25.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:18:25.516 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 100.137901304, 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.517 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.518 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:18:25.520 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.521 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 100.105854307, 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.523 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.525 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.528 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 100.093642354, 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:25.529 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.530 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.531 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.532 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.533 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 100.05974639599999, 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.534 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:25.535 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:18:25.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.537 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.538 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 100.020701527, 2024-09-21T15:18:25.539 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:25.540 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:18:25.541 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.542 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:18:25.543 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.544 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.160258400000004, 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:25.545 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.546 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:18:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:18:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.148680893000005, 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:18:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:18:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:18:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 88.114270125000004, 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:25.561 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:25.563 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:25.563+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:25.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:25.596 INFO:teuthology.orchestra.run.smithi067.stdout:moved 0 / 34 (0%) 2024-09-21T15:18:25.596 INFO:teuthology.orchestra.run.smithi067.stdout:avg 5.66667 2024-09-21T15:18:25.596 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 1.10554 -> 1.10554 (expected baseline 2.17307) 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.5 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.3 with 7 -> 7 pgs (1.23529 -> 1.23529 * mean) 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:oload 1000 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 0.05 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0008 2024-09-21T15:18:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0075 2024-09-21T15:18:25.600 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:18:25.611 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T15:18:25.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 1.0 2024-09-21T15:18:25.664 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:25.663+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:25.675 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:25.676 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:25.676 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:25.676 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:25.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:25.765 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:25.762+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:25.786 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:25.786 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:25.786 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:25.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:25.866 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:25.866+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:25.954 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:25.954 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:25.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:25.954 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:25.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.955 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:25.955 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:25.955 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.493833983, 2024-09-21T15:18:25.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:18:25.955 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:25.956 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.956 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:25.956 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:25.956 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.956 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:18:25.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.957 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:25.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:25.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:25.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:25.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:25.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:25.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:18:25.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:25.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:25.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:18:25.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:18:25.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:25.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:18:25.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:18:25.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:18:25.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:18:25.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:25.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:18:25.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:18:25.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:25.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:18:25.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:18:25.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:18:25.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:18:25.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:18:25.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:18:25.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:25.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:18:25.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:18:25.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:25.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:18:25.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:18:25.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:25.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:18:25.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:18:25.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.967 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:25.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.968 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:25.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.44594972599999, 2024-09-21T15:18:25.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:18:25.969 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:25.969 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.969 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:25.969 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:25.969 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.969 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:18:25.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.970 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:25.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:25.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:25.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:25.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:25.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:25.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:18:25.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:25.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:25.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:18:25.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:18:25.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:25.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:18:25.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:18:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:18:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:18:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:25.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:18:25.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:18:25.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:25.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:18:25.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:18:25.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:18:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:18:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:18:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:18:25.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:18:25.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:18:25.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:25.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:18:25.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:18:25.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.980 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.39777788000001, 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:18:25.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:25.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:18:25.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:18:25.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.039608778, 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:25.987 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:18:25.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:18:25.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:18:25.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:25.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:18:25.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:18:25.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:25.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:18:25.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:18:25.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:25.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:25.994 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.87362490699999, 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:18:25.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:18:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:18:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:18:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:18:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:18:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.688119638, 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:18:26.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:18:26.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:18:26.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:18:26.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:18:26.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:18:26.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:18:26.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:18:26.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.008 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:26.009 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:26.009 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.709480807000006, 2024-09-21T15:18:26.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:18:26.009 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.009 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.009 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.010 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:18:26.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.010 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:18:26.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:18:26.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:18:26.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:18:26.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:18:26.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:18:26.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:18:26.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:18:26.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:18:26.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:18:26.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:18:26.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:18:26.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:18:26.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:18:26.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:18:26.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:18:26.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:18:26.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:18:26.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:18:26.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:18:26.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:18:26.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.020 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.021 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:26.021 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:26.021 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.708695930000005, 2024-09-21T15:18:26.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:18:26.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.022 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.022 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.022 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.022 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.022 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:18:26.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.022 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:26.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:18:26.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.708155805999994, 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.029 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.034 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.70795339, 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:18:26.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.036 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:26.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:26.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:18:26.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.707757185000006, 2024-09-21T15:18:26.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:18:26.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:18:26.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.707563062000006, 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:18:26.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:18:26.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:18:26.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:26.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.653115841000002, 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:18:26.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:18:26.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:18:26.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:18:26.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.652983394000003, 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.060 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:18:26.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.061 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:26.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.652832015000001, 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.066 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:18:26.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:18:26.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.652704643999996, 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.072 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:18:26.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.517904643999998, 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.078 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:18:26.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:18:26.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:18:26.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:18:26.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:18:26.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:26.084 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.380878559999999, 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:18:26.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:18:26.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:18:26.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:18:26.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.205581656000007, 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.092 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:26.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:18:26.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:18:26.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:18:26.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:18:26.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.098 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.155289131000004, 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:18:26.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:18:26.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.104 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:26.105 INFO:teuthology.orchestra.run.smithi067.stdout:moved 0 / 34 (0%) 2024-09-21T15:18:26.105 INFO:teuthology.orchestra.run.smithi067.stdout:avg 5.66667 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 1.10554 -> 1.10554 (expected baseline 2.17307) 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.5 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.3 with 7 -> 7 pgs (1.23529 -> 1.23529 * mean) 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:oload 1000 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 1 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0008 2024-09-21T15:18:26.106 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0075 2024-09-21T15:18:26.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:26.108 INFO:teuthology.orchestra.run.smithi067.stderr:no change 2024-09-21T15:18:26.108 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T15:18:26.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-09-21T15:18:26.108 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:26.106+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:26.206 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:26.206+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:26.285 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 113/112 [PushOp(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head, version: 50'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:18:26.277218+0000", 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.010345596, 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010377769, 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277218+0000", 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277218+0000", 2024-09-21T15:18:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277216+0000", 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277844+0000", 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062720699999999996 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277871+0000", 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7206e-05 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277891+0000", 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0543e-05 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277963+0000", 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1371999999999994e-05 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277972+0000", 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0599999999999997e-06 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 1 2024-09-21T15:18:26.289 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:26.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:26.307 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:26.306+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:26.396 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:26.396 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:26.396 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:26.397 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:26.397 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:26.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:26.450 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:26.450+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:26.509 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:26.507+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:26.562 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4198.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 188.77936415799999, 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053420756, 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4198", 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.142:0/1169731978", 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782221+0000", 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782218+0000", 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782235+0000", 2024-09-21T15:18:26.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6212999999999999e-05 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782258+0000", 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3428999999999999e-05 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782279+0000", 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0713e-05 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782369+0000", 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9765000000000005e-05 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.782388+0000", 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9097e-05 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832031+0000", 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049643038 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.832512+0000", 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048103600000000001 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.833104+0000", 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059266400000000002 2024-09-21T15:18:26.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835393+0000", 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022884369999999999 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835497+0000", 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104614 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835582+0000", 2024-09-21T15:18:26.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4418000000000004e-05 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:17.835642+0000", 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9964999999999999e-05 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.570 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:626 2.7 e57/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 57'35, mlcod=57'33)", 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.102426936, 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035732360999999997, 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459158+0000", 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459156+0000", 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459639+0000", 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048296299999999999 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459669+0000", 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9848e-05 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459683+0000", 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3594000000000001e-05 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459743+0000", 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9772000000000003e-05 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459772+0000", 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9308000000000001e-05 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494767+0000", 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034995327999999999 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.494891+0000", 2024-09-21T15:18:26.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-09-21T15:18:26.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.575 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.575 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.576 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:642 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:26.576 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:26.576 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.774407567, 2024-09-21T15:18:26.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042332346, 2024-09-21T15:18:26.576 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.576 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 642 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787178+0000", 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.787176+0000", 2024-09-21T15:18:26.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788200+0000", 2024-09-21T15:18:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001024795 2024-09-21T15:18:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788232+0000", 2024-09-21T15:18:26.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1748999999999997e-05 2024-09-21T15:18:26.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788247+0000", 2024-09-21T15:18:26.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5356999999999999e-05 2024-09-21T15:18:26.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788349+0000", 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010131700000000001 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788504+0000", 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015564299999999999 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.788699+0000", 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194883 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.816983+0000", 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028283484000000001 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829362+0000", 2024-09-21T15:18:26.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012379324000000001 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829439+0000", 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6860999999999995e-05 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.829510+0000", 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0970000000000007e-05 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:644 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.723246124, 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037223348000000003, 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:26.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838339+0000", 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.838338+0000", 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839094+0000", 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00075636599999999996 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839116+0000", 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1665e-05 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839129+0000", 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.326e-05 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839181+0000", 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2413000000000001e-05 2024-09-21T15:18:26.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839262+0000", 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0125999999999998e-05 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.839375+0000", 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011310400000000001 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.864182+0000", 2024-09-21T15:18:26.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024807578 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875400+0000", 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011217267 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875481+0000", 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1150999999999996e-05 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.875562+0000", 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1698000000000003e-05 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:648 2.3 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.635200158, 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043466433999999998, 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 648 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:26.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926385+0000", 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.926384+0000", 2024-09-21T15:18:26.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927206+0000", 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082232500000000005 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927230+0000", 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3890999999999999e-05 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927240+0000", 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0360000000000001e-05 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927312+0000", 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1132999999999998e-05 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927396+0000", 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4625999999999999e-05 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.927518+0000", 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012203500000000001 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.950680+0000", 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023161607000000001 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969740+0000", 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019060677000000002 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969790+0000", 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9879000000000001e-05 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.969851+0000", 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1143999999999996e-05 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.597 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:676 2.2 e58/48 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head v 58'9, mlcod=58'7)", 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.05466837900001, 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038349304000000001, 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506917+0000", 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506916+0000", 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507294+0000", 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000377981 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507311+0000", 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7410000000000001e-05 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507320+0000", 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7700000000000007e-06 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507395+0000", 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5544999999999996e-05 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.507426+0000", 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0732999999999999e-05 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545193+0000", 2024-09-21T15:18:26.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037767060999999998 2024-09-21T15:18:26.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545266+0000", 2024-09-21T15:18:26.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2857999999999999e-05 2024-09-21T15:18:26.602 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4996.0:696 2.7 e58/56 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.648423697, 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037000953000000003, 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913161+0000", 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913160+0000", 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913596+0000", 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043578600000000002 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913623+0000", 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7157000000000001e-05 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913638+0000", 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4952e-05 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913703+0000", 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5159000000000004e-05 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913732+0000", 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8963e-05 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950126+0000", 2024-09-21T15:18:26.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036393322999999998 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950162+0000", 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6910999999999999e-05 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:723 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.085639927, 2024-09-21T15:18:26.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037985894999999999, 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 723 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475945+0000", 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.475943+0000", 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476921+0000", 2024-09-21T15:18:26.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00097707099999999993 2024-09-21T15:18:26.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476941+0000", 2024-09-21T15:18:26.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-09-21T15:18:26.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.476952+0000", 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1022000000000001e-05 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477051+0000", 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8888000000000006e-05 2024-09-21T15:18:26.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477166+0000", 2024-09-21T15:18:26.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011542799999999999 2024-09-21T15:18:26.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.477320+0000", 2024-09-21T15:18:26.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154069 2024-09-21T15:18:26.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.512603+0000", 2024-09-21T15:18:26.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035282925 2024-09-21T15:18:26.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513798+0000", 2024-09-21T15:18:26.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011947990000000001 2024-09-21T15:18:26.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513853+0000", 2024-09-21T15:18:26.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4595999999999997e-05 2024-09-21T15:18:26.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.513931+0000", 2024-09-21T15:18:26.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8658000000000005e-05 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:729 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.96081187999999, 2024-09-21T15:18:26.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037081293000000001, 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 729 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600773+0000", 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.600771+0000", 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601630+0000", 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085884799999999995 2024-09-21T15:18:26.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601672+0000", 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1436000000000001e-05 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601693+0000", 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1616000000000001e-05 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601765+0000", 2024-09-21T15:18:26.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2189999999999998e-05 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.601872+0000", 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010625599999999999 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.602037+0000", 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016549499999999999 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.636766+0000", 2024-09-21T15:18:26.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034728319000000001 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637626+0000", 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086057699999999996 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637754+0000", 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012785300000000001 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.637855+0000", 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010061200000000001 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:59 2.7 e62/56 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head v 62'114, mlcod=62'112)", 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.814321379000006, 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037668802000000001, 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747264+0000", 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747263+0000", 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747650+0000", 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038735700000000001 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747670+0000", 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0531e-05 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747681+0000", 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0883999999999999e-05 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747743+0000", 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2131999999999994e-05 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747774+0000", 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0422000000000001e-05 2024-09-21T15:18:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784864+0000", 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037090515999999997 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.784933+0000", 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8232999999999995e-05 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:64 2.7 e62/56 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head v 62'119, mlcod=62'117)", 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.718015957000006, 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036809112999999997, 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843569+0000", 2024-09-21T15:18:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.843567+0000", 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844073+0000", 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050527099999999995 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844092+0000", 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9627999999999999e-05 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844101+0000", 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6929999999999996e-06 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844161+0000", 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0170999999999997e-05 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.844193+0000", 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1705999999999998e-05 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880326+0000", 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036132574000000001 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.880378+0000", 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2828000000000003e-05 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:170 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 89.167845485000001, 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044479754000000003, 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.632 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 170 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393740+0000", 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.393738+0000", 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394787+0000", 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010488489999999999 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394814+0000", 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7676999999999999e-05 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394827+0000", 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2707e-05 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.394927+0000", 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9933999999999996e-05 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395085+0000", 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015823599999999999 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395316+0000", 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023038900000000001 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.437462+0000", 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042146649000000001 2024-09-21T15:18:26.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438153+0000", 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069098599999999999 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438189+0000", 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6267999999999999e-05 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.438219+0000", 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0006e-05 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:172 2.3 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 89.115006961000006, 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038538415999999999, 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.638 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 172 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446578+0000", 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.446573+0000", 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999943 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447433+0000", 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086085800000000004 2024-09-21T15:18:26.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447447+0000", 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3412e-05 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447453+0000", 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.241e-06 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447494+0000", 2024-09-21T15:18:26.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1182e-05 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447562+0000", 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7288999999999999e-05 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447645+0000", 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3305000000000003e-05 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:26.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.472537+0000", 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024892234999999999 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484971+0000", 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012433516 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485054+0000", 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3870999999999997e-05 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.485117+0000", 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:200 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'117, mlcod=76'112)", 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.237075283999999, 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037395221999999999, 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.645 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324510+0000", 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324509+0000", 2024-09-21T15:18:26.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324564+0000", 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4592999999999997e-05 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324578+0000", 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4307e-05 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.324585+0000", 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8680000000000001e-06 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326616+0000", 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020308629999999999 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.326653+0000", 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7857e-05 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361835+0000", 2024-09-21T15:18:26.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035181104999999997 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.361905+0000", 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0593000000000007e-05 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:202 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'119, mlcod=76'113)", 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.229728547999997, 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035953194000000001, 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331857+0000", 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331856+0000", 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331968+0000", 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112661 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331983+0000", 2024-09-21T15:18:26.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4739999999999999e-05 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.331990+0000", 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1550000000000001e-06 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341419+0000", 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094287910000000006 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.341449+0000", 2024-09-21T15:18:26.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9620000000000001e-05 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367753+0000", 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026304079000000001 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.367810+0000", 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7055000000000002e-05 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5557.0:286 2.5 e77/68 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head v 77'124, mlcod=77'122)", 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 69.054693705999995, 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038020446999999999, 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506891+0000", 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.506889+0000", 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508309+0000", 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014198920000000001 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508358+0000", 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9781999999999999e-05 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508381+0000", 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2826000000000001e-05 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508459+0000", 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7998999999999996e-05 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.508491+0000", 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1222e-05 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544853+0000", 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036362951999999997 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:17.544912+0000", 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8446999999999998e-05 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 113/112 [PushOp(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head, version: 50'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:18:26.233316+0000", 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.328268909, 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037497175000000001, 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.233316+0000", 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.233316+0000", 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.233315+0000", 2024-09-21T15:18:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.234193+0000", 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087745399999999995 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.234214+0000", 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1455e-05 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.234227+0000", 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2734e-05 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.234321+0000", 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4121000000000002e-05 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.234334+0000", 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-05 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.270813+0000", 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036479770000000002 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 113/112 [PushOp(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head, version: 50'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:18:26.277218+0000", 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.28436768200000001, 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037284192000000001, 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277218+0000", 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277218+0000", 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277216+0000", 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277844+0000", 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00062720699999999996 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277871+0000", 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7206e-05 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277891+0000", 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0543e-05 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277963+0000", 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1371999999999994e-05 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.277972+0000", 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0599999999999997e-06 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.314502+0000", 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036529903000000002 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 113/112 [PushOp(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4c676614:ns::rbd_data.137aae78387e.0000000000000001:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:18:26.318516+0000", 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.243069166, 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036522708000000001, 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.666 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.318516+0000", 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.318516+0000", 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.318515+0000", 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.319323+0000", 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00080810699999999997 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.319355+0000", 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2737000000000002e-05 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.319386+0000", 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0485e-05 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.319481+0000", 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5414000000000005e-05 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.319499+0000", 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7589999999999999e-05 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.355039+0000", 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03553986 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(2.2 113/112 [PushOp(2:5074958c:ns::rbd_data.137aae78387e.0000000000000005:head, version: 50'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:5074958c:ns::rbd_data.137aae78387e.0000000000000005:head@50'5, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:18:26.395506+0000", 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.16607941800000001, 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043787868000000001, 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:26.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.395506+0000", 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.395506+0000", 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.395505+0000", 2024-09-21T15:18:26.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.396094+0000", 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00058989299999999997 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.396111+0000", 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6855e-05 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:26.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.396121+0000", 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8879999999999992e-06 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.396211+0000", 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0075999999999996e-05 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.396227+0000", 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6155e-05 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:26.439294+0000", 2024-09-21T15:18:26.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043066225 2024-09-21T15:18:26.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:26.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:26.674 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:26.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:18:26.676 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:26.675+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:26.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:26.776 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:26.775+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:26.784 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:26.784 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:26.784 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:26.784 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:26.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:26.813 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:26.838 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:26.856 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:26.877 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:26.874+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:26.878 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:26.899 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:26.900 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:26.906 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:26.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:26.935 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:26.937 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:26.937 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:26.937 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:26.938 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:26.938 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:26.938 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:26.979 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:26.975+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:27.016 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.016 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:27.079 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:27.078+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:27.143 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.143 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.153 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:27.180 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:27.178+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:27.224 INFO:teuthology.orchestra.run.smithi067.stdout:moved 4 / 34 (11.7647%) 2024-09-21T15:18:27.224 INFO:teuthology.orchestra.run.smithi067.stdout:avg 5.66667 2024-09-21T15:18:27.224 INFO:teuthology.orchestra.run.smithi067.stdout:stddev 1.10554 -> 2.35702 (expected baseline 2.17307) 2024-09-21T15:18:27.224 INFO:teuthology.orchestra.run.smithi067.stdout:min osd.5 with 4 -> 4 pgs (0.705882 -> 0.705882 * mean) 2024-09-21T15:18:27.224 INFO:teuthology.orchestra.run.smithi067.stdout:max osd.3 with 7 -> 8 pgs (1.23529 -> 1.41176 * mean) 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout:oload 110 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout:max_change 1 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout:max_change_osds 4 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout:average_utilization 0.0008 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout:overload_utilization 0.0008 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 weight 0.6964 -> 0.4445 2024-09-21T15:18:27.225 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 weight 0.8722 -> 0.6971 2024-09-21T15:18:27.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:18:27.238 INFO:tasks.workunit.client.0.smithi146.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete...2024-09-21T15:18:27.237+0000 7fae382d8700 0 -- 172.21.15.146:0/4040879226 >> [v2:172.21.15.142:6816/3606282773,v1:172.21.15.142:6817/3606282773] conn(0x7fae1c008be0 msgr2=0x7fae1c00b070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:18:27.242 INFO:tasks.workunit.client.0.smithi146.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T15:18:27.259 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.259 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.269 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:27.286 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:27.286+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:27.371 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.371 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:27.381 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:27.378+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:27.482 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:27.479+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:27.486 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.486 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.495 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:27.582 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:27.582+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:27.603 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.604 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:27.683 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:27.679+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:27.724 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.724 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.733 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:27.784 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:27.779+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:27.885 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:27.883+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:27.889 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:27.889 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:27.899 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:27.985 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:27.982+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:28.052 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.052 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:28.086 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:28.083+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:28.168 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.168 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:28.226 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:28.226+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:18:28.288 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:28.283+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:28.312 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.312 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:28.389 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:28.386+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:28.428 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.428 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:28.489 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:28.486+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:28.552 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.552 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:28.606 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:28.606+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:28.691 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:28.687+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:28.716 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.716 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:28.793 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:28.791+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:28.829 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.830 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:28.893 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:28.891+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:28.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:18:28.952 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:28.952 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:28.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:28.994 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:28.991+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:29.069 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:29.069 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:29.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:29.087 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:29.095 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:29.094+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:29.106 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:29.124 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:29.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:29.169 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:29.178 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:29.178 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:29.178 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:29.178 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:29.178 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:29.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:29.192 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:29.195 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:29.191+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:29.244 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd17425 2024-09-21T15:18:29.296 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:29.294+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:29.308 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4208.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 186.485510629, 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002473954, 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822362+0000", 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822361+0000", 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822378+0000", 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6453999999999999e-05 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822458+0000", 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0366999999999998e-05 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822475+0000", 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6909999999999999e-05 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822564+0000", 2024-09-21T15:18:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8991000000000002e-05 2024-09-21T15:18:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.822615+0000", 2024-09-21T15:18:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0466000000000001e-05 2024-09-21T15:18:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824777+0000", 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021619899999999999 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:15:22.824836+0000", 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9308000000000002e-05 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:169 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'32, mlcod=63'30)", 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.923086127000005, 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008109883, 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384786+0000", 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384849+0000", 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2762000000000007e-05 2024-09-21T15:18:29.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384862+0000", 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3536000000000001e-05 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384870+0000", 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1300000000000001e-06 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384943+0000", 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2897000000000004e-05 2024-09-21T15:18:29.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.384986+0000", 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2358999999999999e-05 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392785+0000", 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077997320000000002 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.392896+0000", 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110911 2024-09-21T15:18:29.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:170 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'33, mlcod=63'31)", 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.912349625999994, 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038395313, 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:29.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395523+0000", 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.395521+0000", 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396293+0000", 2024-09-21T15:18:29.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00077178399999999997 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396322+0000", 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8668999999999999e-05 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396337+0000", 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5109999999999999e-05 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396453+0000", 2024-09-21T15:18:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116073 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.396526+0000", 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3199000000000002e-05 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433805+0000", 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037278525 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.433918+0000", 2024-09-21T15:18:29.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113511 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:171 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'34, mlcod=64'32)", 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.867892792000006, 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058898680000000004, 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439980+0000", 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.439978+0000", 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440053+0000", 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5082999999999999e-05 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440076+0000", 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2932999999999999e-05 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440087+0000", 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0832999999999999e-05 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440154+0000", 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.440181+0000", 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7373e-05 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445833+0000", 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056518760000000001 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.445870+0000", 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6740999999999999e-05 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:172 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'35, mlcod=64'33)", 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.860115989999997, 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037041019000000001, 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447757+0000", 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.447755+0000", 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448114+0000", 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035926399999999998 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448140+0000", 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5459000000000001e-05 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448150+0000", 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0677e-05 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448210+0000", 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.448240+0000", 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0772999999999998e-05 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484688+0000", 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036447809999999997 2024-09-21T15:18:29.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.484798+0000", 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109392 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:173 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'36, mlcod=64'34)", 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.820942318999997, 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040706340000000001, 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.331 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486930+0000", 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.486929+0000", 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487069+0000", 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014049399999999999 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487091+0000", 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1177000000000001e-05 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487100+0000", 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6539999999999994e-06 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487162+0000", 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2080000000000002e-05 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.487190+0000", 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999999e-05 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.490935+0000", 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037453730000000002 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.491001+0000", 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5767999999999998e-05 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:174 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'37, mlcod=64'35)", 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:29.335 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.814209958000006, 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031653696000000002, 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493663+0000", 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493662+0000", 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493937+0000", 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027537999999999998 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493955+0000", 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7550999999999998e-05 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.493962+0000", 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7810000000000005e-06 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494058+0000", 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5602999999999999e-05 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.494096+0000", 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8279999999999999e-05 2024-09-21T15:18:29.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525287+0000", 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031190907 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.525316+0000", 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9158000000000001e-05 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.339 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:175 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'38, mlcod=64'36)", 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.780158466000003, 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040431149999999999, 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527714+0000", 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527713+0000", 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527806+0000", 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3251999999999994e-05 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527836+0000", 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9961999999999999e-05 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527867+0000", 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1139999999999997e-05 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527919+0000", 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2062999999999999e-05 2024-09-21T15:18:29.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.527935+0000", 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6395000000000001e-05 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531667+0000", 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037319559999999998 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.531757+0000", 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9870000000000005e-05 2024-09-21T15:18:29.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5206.0:176 2.3 e64/62 2:da2d6458:::rbd_data.13743413a48b.000000000000000f:head v 64'39, mlcod=64'37)", 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.770032482000005, 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028016242, 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:29.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537840+0000", 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.537839+0000", 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538246+0000", 2024-09-21T15:18:29.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040704199999999998 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538268+0000", 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2660999999999999e-05 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538280+0000", 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1800000000000001e-05 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538325+0000", 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5077000000000002e-05 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.538343+0000", 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8159999999999999e-05 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565769+0000", 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027425456000000001 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:57.565856+0000", 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7558000000000004e-05 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.5490.0:25 2.3 e72/62 2:c5f410d8:ns::rbd_object_map.137aae78387e.0000000000000005:head v 72'40, mlcod=64'38)", 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.073269097999997, 2024-09-21T15:18:29.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086925399999999997, 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234603+0000", 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234610+0000", 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234624+0000", 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4630999999999999e-05 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234633+0000", 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6899999999999998e-06 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234683+0000", 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0331000000000003e-05 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.234702+0000", 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8425000000000001e-05 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235423+0000", 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072120500000000004 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.235473+0000", 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9746000000000001e-05 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:1 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:29.352 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.935889794999994, 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047030599999999998, 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371983+0000", 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371978+0000", 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999957 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.371994+0000", 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5206e-05 2024-09-21T15:18:29.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372018+0000", 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4714000000000001e-05 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372036+0000", 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7286e-05 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372173+0000", 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013774699999999999 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372331+0000", 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015799999999999999 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372453+0000", 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121743 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5502.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.935065742999996, 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000851952, 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5502", 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2230569681", 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372807+0000", 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372805+0000", 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372816+0000", 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-05 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372839+0000", 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699000000000001e-05 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372851+0000", 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1956999999999999e-05 2024-09-21T15:18:29.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.372966+0000", 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011531000000000001 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373215+0000", 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024845200000000003 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:07.373659+0000", 2024-09-21T15:18:29.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044398700000000002 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5508.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.050004195, 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064051500000000003, 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5508", 2024-09-21T15:18:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4136796126", 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257868+0000", 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257866+0000", 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257879+0000", 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2853999999999999e-05 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257902+0000", 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3368000000000001e-05 2024-09-21T15:18:29.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.257922+0000", 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9899999999999999e-05 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258055+0000", 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013332 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258273+0000", 2024-09-21T15:18:29.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000217529 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.258509+0000", 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023605499999999999 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5410.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 80.909372740999999, 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081098400000000005, 2024-09-21T15:18:29.365 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5410", 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3054493933", 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398500+0000", 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398499+0000", 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398507+0000", 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398525+0000", 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8300000000000001e-05 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398555+0000", 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9382000000000001e-05 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398675+0000", 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120253 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.398903+0000", 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000227684 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:08.399311+0000", 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040820500000000003 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.369 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5201.0:2 2.0 2:1249b2b9:ns::rbd_id.testrbdnbd17425:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.615459466000004, 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084375100000000001, 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5201", 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/2910097456", 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692413+0000", 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692412+0000", 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692418+0000", 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0929999999999999e-06 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692428+0000", 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1949999999999992e-06 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692442+0000", 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4069e-05 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692562+0000", 2024-09-21T15:18:29.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012063200000000001 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.692826+0000", 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026331800000000002 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:11.693257+0000", 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043124 2024-09-21T15:18:29.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5557.0:362 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [writefull 0~38] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 63.975520725999999, 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005034312, 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5557", 2024-09-21T15:18:29.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 362 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332352+0000", 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332350+0000", 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332359+0000", 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4299999999999995e-06 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332375+0000", 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5664000000000001e-05 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332386+0000", 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0706999999999999e-05 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332467+0000", 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1440999999999995e-05 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.332473+0000", 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.003e-06 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.334896+0000", 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024229329999999999 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335011+0000", 2024-09-21T15:18:29.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114988 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.335128+0000", 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011657100000000001 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.336158+0000", 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010302779999999999 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337238+0000", 2024-09-21T15:18:29.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010804619999999999 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337334+0000", 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5744999999999999e-05 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:25.337386+0000", 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2061999999999997e-05 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:5cea7035:::rbd_info:head)])", 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.053399906000003, 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049273299999999997, 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254473+0000", 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254471+0000", 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254485+0000", 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4589e-05 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254518+0000", 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2280000000000003e-05 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254540+0000", 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2345999999999999e-05 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254656+0000", 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011644799999999999 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.254965+0000", 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030890300000000002 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:40f9bc9a:::rbd_object_map.158254dca816:head)])", 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:29.384 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.050673541999998, 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000505742, 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257199+0000", 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257198+0000", 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257207+0000", 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.713e-06 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257233+0000", 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6020000000000002e-05 2024-09-21T15:18:29.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257252+0000", 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8953e-05 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257388+0000", 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000135907 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.257705+0000", 2024-09-21T15:18:29.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031644600000000001 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2 92/91 [PushReplyOp(3:59ae3db4:::rbd_header.158254dca816:head)])", 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.048076016000003, 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00054857099999999998, 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259797+0000", 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259796+0000", 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259804+0000", 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259843+0000", 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8976000000000001e-05 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259867+0000", 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.259979+0000", 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111561 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:17:44.260345+0000", 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036630799999999998 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5846.0:38 2.0 2:1197f94a:::rbd_object_map.13743413a48b.0000000000000007:head [call rbd.object_map_snap_remove in=25b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e109)", 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:18:22.208990+0000", 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.0988823639999996, 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023769099999999999, 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5846", 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/4040879226", 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 38 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.208990+0000", 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.208990+0000", 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.208988+0000", 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:29.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.208998+0000", 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4939999999999992e-06 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.209015+0000", 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6903999999999999e-05 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.209025+0000", 2024-09-21T15:18:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0016e-05 2024-09-21T15:18:29.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:29.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.209100+0000", 2024-09-21T15:18:29.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5636999999999994e-05 2024-09-21T15:18:29.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:29.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.209185+0000", 2024-09-21T15:18:29.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4339999999999995e-05 2024-09-21T15:18:29.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-09-21T15:18:29.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.209519+0000", 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033414399999999998 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.210367+0000", 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000848247 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.211262+0000", 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089503300000000005 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.211322+0000", 2024-09-21T15:18:29.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0273999999999999e-05 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:18:22.211367+0000", 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4852999999999999e-05 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:29.400 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:29.400 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:29.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:29.403 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:29.398+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:29.456 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:29.453+0000 7f09fafd8700 0 -- 172.21.15.146:0/2730700777 >> [v2:172.21.15.142:6816/3606282773,v1:172.21.15.142:6817/3606282773] conn(0x55fee8c03200 msgr2=0x55fee8c055f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:18:29.498 INFO:tasks.workunit.client.0.smithi146.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-21T15:18:29.497+0000 7f09fafd8700 0 -- 172.21.15.146:0/2730700777 >> [v2:172.21.15.67:6800/3371454484,v1:172.21.15.67:6801/3371454484] conn(0x55fee8ca22e0 msgr2=0x55fee8c507d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:18:29.498 INFO:tasks.workunit.client.0.smithi146.stderr: Removing image: 100% complete...done. 2024-09-21T15:18:29.502 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:29.499+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:29.506 INFO:tasks.workunit.client.0.smithi146.stderr:+ for ns in '' ${NS} 2024-09-21T15:18:29.506 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace ns status testrbdnbd17425 2024-09-21T15:18:29.514 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:29.514 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:29.514 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:29.515 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:29.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:29.573 INFO:tasks.workunit.client.0.smithi146.stdout:Watchers: none 2024-09-21T15:18:29.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-21T15:18:29.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ sleep 0.5 2024-09-21T15:18:29.603 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:29.602+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:29.629 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:29.629 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:29.630 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:29.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:29.704 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:29.699+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:29.751 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:29.751 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:29.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:29.751 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:29.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.751 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:634 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'37, mlcod=57'35)", 2024-09-21T15:18:29.751 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.12576775300001, 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031398479999999999, 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624713+0000", 2024-09-21T15:18:29.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.624711+0000", 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625638+0000", 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00092671599999999995 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625658+0000", 2024-09-21T15:18:29.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9423e-05 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625669+0000", 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1285e-05 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625775+0000", 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000105942 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.625836+0000", 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1255999999999998e-05 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656018+0000", 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030181669000000001 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.656112+0000", 2024-09-21T15:18:29.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3817e-05 2024-09-21T15:18:29.756 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.756 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.756 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.756 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:640 2.6 e57/48 2:7b070134:::rbd_data.13743413a48b.0000000000000004:head v 57'43, mlcod=57'41)", 2024-09-21T15:18:29.757 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:29.757 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.009619816, 2024-09-21T15:18:29.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031971063000000001, 2024-09-21T15:18:29.757 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.757 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.757 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:29.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740861+0000", 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.740859+0000", 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741736+0000", 2024-09-21T15:18:29.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00087687799999999999 2024-09-21T15:18:29.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741764+0000", 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8232999999999999e-05 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741784+0000", 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9559999999999999e-05 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741833+0000", 2024-09-21T15:18:29.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9181999999999998e-05 2024-09-21T15:18:29.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.741853+0000", 2024-09-21T15:18:29.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0373e-05 2024-09-21T15:18:29.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772748+0000", 2024-09-21T15:18:29.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030894466999999998 2024-09-21T15:18:29.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.772832+0000", 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.407e-05 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:642 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'7, mlcod=50'5)", 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.962999419, 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039983019000000002, 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:29.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787481+0000", 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.787479+0000", 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788594+0000", 2024-09-21T15:18:29.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011146070000000001 2024-09-21T15:18:29.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788624+0000", 2024-09-21T15:18:29.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0196999999999999e-05 2024-09-21T15:18:29.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788644+0000", 2024-09-21T15:18:29.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9723000000000001e-05 2024-09-21T15:18:29.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.770 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788704+0000", 2024-09-21T15:18:29.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0012000000000001e-05 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.788733+0000", 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9272e-05 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827403+0000", 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038669424000000001 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.827464+0000", 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.1654000000000003e-05 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:644 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'9, mlcod=57'7)", 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.912382795, 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035417351999999999, 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.773 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:29.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838098+0000", 2024-09-21T15:18:29.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838097+0000", 2024-09-21T15:18:29.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.776 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.776 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838851+0000", 2024-09-21T15:18:29.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00075436500000000001 2024-09-21T15:18:29.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838869+0000", 2024-09-21T15:18:29.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:18:29.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838877+0000", 2024-09-21T15:18:29.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.1640000000000001e-06 2024-09-21T15:18:29.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838910+0000", 2024-09-21T15:18:29.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2991999999999999e-05 2024-09-21T15:18:29.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.838925+0000", 2024-09-21T15:18:29.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5315000000000002e-05 2024-09-21T15:18:29.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873378+0000", 2024-09-21T15:18:29.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.034453224999999997 2024-09-21T15:18:29.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.873515+0000", 2024-09-21T15:18:29.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013685999999999999 2024-09-21T15:18:29.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.781 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.782 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:648 2.3 e57/48 2:d37b711d:::rbd_data.13743413a48b.0000000000000005:head v 57'13, mlcod=57'11)", 2024-09-21T15:18:29.782 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:29.782 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.824029323, 2024-09-21T15:18:29.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041485608, 2024-09-21T15:18:29.782 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.782 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.783 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:29.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926451+0000", 2024-09-21T15:18:29.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.926450+0000", 2024-09-21T15:18:29.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927235+0000", 2024-09-21T15:18:29.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00078502800000000001 2024-09-21T15:18:29.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927264+0000", 2024-09-21T15:18:29.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.889e-05 2024-09-21T15:18:29.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927277+0000", 2024-09-21T15:18:29.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3006e-05 2024-09-21T15:18:29.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927326+0000", 2024-09-21T15:18:29.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9579000000000001e-05 2024-09-21T15:18:29.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.927351+0000", 2024-09-21T15:18:29.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5318999999999999e-05 2024-09-21T15:18:29.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967843+0000", 2024-09-21T15:18:29.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040491679000000003 2024-09-21T15:18:29.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:43.967937+0000", 2024-09-21T15:18:29.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3953e-05 2024-09-21T15:18:29.790 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.790 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.790 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:672 2.4 e58/17 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head v 58'16, mlcod=58'14)", 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.33489845299999, 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032528779000000001, 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:29.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415582+0000", 2024-09-21T15:18:29.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415581+0000", 2024-09-21T15:18:29.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:29.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415996+0000", 2024-09-21T15:18:29.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041450699999999999 2024-09-21T15:18:29.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416010+0000", 2024-09-21T15:18:29.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3993e-05 2024-09-21T15:18:29.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416016+0000", 2024-09-21T15:18:29.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2260000000000002e-06 2024-09-21T15:18:29.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416096+0000", 2024-09-21T15:18:29.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0098000000000005e-05 2024-09-21T15:18:29.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.416132+0000", 2024-09-21T15:18:29.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6053000000000002e-05 2024-09-21T15:18:29.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448079+0000", 2024-09-21T15:18:29.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031946545 2024-09-21T15:18:29.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.798 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.798 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448111+0000", 2024-09-21T15:18:29.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.2295999999999997e-05 2024-09-21T15:18:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:706 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'53, mlcod=58'51)", 2024-09-21T15:18:29.800 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:29.800 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.62001761099999, 2024-09-21T15:18:29.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031290974999999999, 2024-09-21T15:18:29.800 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.800 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.800 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130463+0000", 2024-09-21T15:18:29.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.130462+0000", 2024-09-21T15:18:29.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:29.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131274+0000", 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081225700000000004 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131293+0000", 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9049999999999999e-05 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131301+0000", 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6539999999999999e-06 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.804 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131425+0000", 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012379200000000001 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.131462+0000", 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6872999999999997e-05 2024-09-21T15:18:29.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161677+0000", 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030215760000000001 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.161754+0000", 2024-09-21T15:18:29.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6736000000000005e-05 2024-09-21T15:18:29.807 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.807 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.807 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.808 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.4996.0:710 2.6 e58/48 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head v 58'57, mlcod=58'55)", 2024-09-21T15:18:29.808 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:29.808 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 104.531703354, 2024-09-21T15:18:29.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031625042999999999, 2024-09-21T15:18:29.808 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.808 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.809 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:29.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218777+0000", 2024-09-21T15:18:29.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.218776+0000", 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219668+0000", 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00089226799999999999 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219695+0000", 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219713+0000", 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7249e-05 2024-09-21T15:18:29.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219788+0000", 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.5646e-05 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.219825+0000", 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:18:29.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250286+0000", 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030461185000000002 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.250402+0000", 2024-09-21T15:18:29.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116455 2024-09-21T15:18:29.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.815 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.815 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_repop(client.5206.0:167 2.1 e64/22 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head v 64'20, mlcod=64'18)", 2024-09-21T15:18:29.815 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:29.816 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 92.412357747000001, 2024-09-21T15:18:29.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036469586999999998, 2024-09-21T15:18:29.816 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.816 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:29.816 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:29.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338123+0000", 2024-09-21T15:18:29.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338121+0000", 2024-09-21T15:18:29.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338689+0000", 2024-09-21T15:18:29.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056799200000000002 2024-09-21T15:18:29.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338712+0000", 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2809000000000001e-05 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338728+0000", 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5874000000000001e-05 2024-09-21T15:18:29.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338786+0000", 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8094000000000003e-05 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.338808+0000", 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1662e-05 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374557+0000", 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.035748873 2024-09-21T15:18:29.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374593+0000", 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5858000000000001e-05 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head, version: 50'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:39fac1e4:ns::rbd_data.137aae78387e.0000000000000000:head@50'4, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:29.822 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 46.200215780000001, 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031117406, 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550265+0000", 2024-09-21T15:18:29.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.550263+0000", 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551427+0000", 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0011644310000000001 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551465+0000", 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7101999999999998e-05 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551492+0000", 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7736e-05 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551635+0000", 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000142686 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.551651+0000", 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5877000000000001e-05 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.581382+0000", 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029731530999999999 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head, version: 50'6, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2b691efa:ns::rbd_data.137aae78387e.000000000000000c:head@50'6, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.938011908, 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030336687000000001, 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812469+0000", 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812467+0000", 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812888+0000", 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00042071099999999999 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812927+0000", 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8733000000000003e-05 2024-09-21T15:18:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.812956+0000", 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9416000000000001e-05 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813137+0000", 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018029 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.813173+0000", 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6761999999999998e-05 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.842806+0000", 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029632209 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.904961950000001, 2024-09-21T15:18:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033027358999999999, 2024-09-21T15:18:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845519+0000", 2024-09-21T15:18:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845518+0000", 2024-09-21T15:18:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845930+0000", 2024-09-21T15:18:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041154899999999999 2024-09-21T15:18:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845951+0000", 2024-09-21T15:18:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:18:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.845969+0000", 2024-09-21T15:18:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8105e-05 2024-09-21T15:18:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846090+0000", 2024-09-21T15:18:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120982 2024-09-21T15:18:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.846111+0000", 2024-09-21T15:18:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0896000000000001e-05 2024-09-21T15:18:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.878546+0000", 2024-09-21T15:18:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032435429000000002 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head, version: 50'7, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2330b941:ns::rbd_data.137aae78387e.000000000000000d:head@50'7, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.816104674000002, 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033417609000000001, 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934376+0000", 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934375+0000", 2024-09-21T15:18:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934912+0000", 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00053620600000000001 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934926+0000", 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4594e-05 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.934935+0000", 2024-09-21T15:18:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2690000000000005e-06 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935057+0000", 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012180600000000001 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.935077+0000", 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:43.967794+0000", 2024-09-21T15:18:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032716325999999997 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.746851208000002, 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039918156000000003, 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003630+0000", 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.003628+0000", 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004235+0000", 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00060648400000000002 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004251+0000", 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6433e-05 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004266+0000", 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4705999999999999e-05 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004380+0000", 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000113792 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.004396+0000", 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6767000000000001e-05 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.043548+0000", 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039151274 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.70296364, 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.041164259000000002, 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047517+0000", 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.047515+0000", 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048536+0000", 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010211389999999999 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048577+0000", 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0759000000000001e-05 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048608+0000", 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1083e-05 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048699+0000", 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1532000000000004e-05 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.048712+0000", 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2996000000000001e-05 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.088681+0000", 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.039969012999999998 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head, version: 50'8, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e98e259:ns::rbd_data.137aae78387e.000000000000000e:head@50'8, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.658082931000003, 2024-09-21T15:18:29.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031092103999999999, 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092398+0000", 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.092396+0000", 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093419+0000", 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010223459999999999 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093457+0000", 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8062000000000003e-05 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093488+0000", 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1123e-05 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093681+0000", 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00019298300000000001 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.093703+0000", 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2036e-05 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.123490+0000", 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029787246999999999 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.4 92/91 [PushOp(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7, version: 85'26, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2e35f377:::rbd_data.13743413a48b.0000000000000008:7@85'26, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 7=[]:{7=[7,6,4]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 45.565652356000001, 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031527579, 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.853 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184828+0000", 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.184827+0000", 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:29.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185868+0000", 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001041132 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185905+0000", 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7406000000000001e-05 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.185939+0000", 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3501999999999999e-05 2024-09-21T15:18:29.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186095+0000", 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015609399999999999 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.186117+0000", 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2269999999999999e-05 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:44.216356+0000", 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030238984999999999 2024-09-21T15:18:29.856 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.7 116/115 [PushOp(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head, version: 50'24, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head@50'24, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:18:29.236838+0000", 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.51364279199999996, 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045464272999999999, 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.236838+0000", 2024-09-21T15:18:29.857 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.236838+0000", 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.236837+0000", 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238665+0000", 2024-09-21T15:18:29.858 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001828447 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238708+0000", 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3105000000000001e-05 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238733+0000", 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4437e-05 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238848+0000", 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011513100000000001 2024-09-21T15:18:29.859 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238860+0000", 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2218999999999999e-05 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.282302+0000", 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.043441987000000001 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.860 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.7 116/115 [PushOp(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head, version: 50'24, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head@50'24, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:18:29.285193+0000", 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.46528752000000001, 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030355995, 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.285193+0000", 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.861 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.285193+0000", 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.285192+0000", 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.286193+0000", 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001000578 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.862 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.286218+0000", 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5723000000000001e-05 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.286235+0000", 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6518e-05 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.286305+0000", 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9806000000000002e-05 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.286317+0000", 2024-09-21T15:18:29.863 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.315549+0000", 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029232131000000001 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.7 116/115 [PushOp(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head, version: 50'24, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head@50'24, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:18:29.353969+0000", 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 0.39651186100000002, 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033705321000000003, 2024-09-21T15:18:29.864 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.353969+0000", 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.353969+0000", 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:29.865 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.353968+0000", 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.354930+0000", 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00096238900000000002 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.354950+0000", 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0125000000000001e-05 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:29.866 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.354964+0000", 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3755e-05 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.355095+0000", 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013110199999999999 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.355123+0000", 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7834000000000002e-05 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:29.867 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.387674+0000", 2024-09-21T15:18:29.868 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032551327999999997 2024-09-21T15:18:29.868 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.868 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.868 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.868 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:29.868 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:29.868 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:29.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:29.870 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:29.867+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:29.970 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:29.967+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:29.987 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:29.987 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:29.987 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:29.987 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:29.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:30.071 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:30.070+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:30.081 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace ns status testrbdnbd17425 2024-09-21T15:18:30.081 INFO:tasks.workunit.client.0.smithi146.stderr:+ grep 'Watchers: none' 2024-09-21T15:18:30.149 INFO:tasks.workunit.client.0.smithi146.stdout:Watchers: none 2024-09-21T15:18:30.150 INFO:tasks.workunit.client.0.smithi146.stderr:+ break 2024-09-21T15:18:30.151 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd17425 2024-09-21T15:18:30.158 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:30.158 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:30.158 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:30.158 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:30.158 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:30.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:30.171 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:30.167+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:30.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:18:30.287 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:30.287 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:30.287 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:30.287 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:30.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 67/66 [PushOp(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head, version: 62'108, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7b070134:::rbd_data.13743413a48b.0000000000000004:head@62'108, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 87.021723078999997, 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046627887999999999, 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262872+0000", 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.262870+0000", 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264070+0000", 2024-09-21T15:18:30.289 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001200223 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264094+0000", 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.3507999999999999e-05 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264109+0000", 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4931e-05 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.290 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264205+0000", 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6298999999999994e-05 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.264223+0000", 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8084000000000002e-05 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.309500+0000", 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045276423000000003 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.291 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.937098423999998, 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.056895836999999998, 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347496+0000", 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.347495+0000", 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349000+0000", 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0015043610000000001 2024-09-21T15:18:30.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349027+0000", 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7656000000000001e-05 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.349048+0000", 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0343000000000002e-05 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375872+0000", 2024-09-21T15:18:30.294 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026824911 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.375899+0000", 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6889e-05 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.404392+0000", 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028492911999999999 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.874791418000001, 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.057724493000000002, 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.296 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409803+0000", 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.409802+0000", 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411826+0000", 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0020234960000000001 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.297 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411854+0000", 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8445999999999998e-05 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.411878+0000", 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4264999999999999e-05 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438049+0000", 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026170695000000001 2024-09-21T15:18:30.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.438071+0000", 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1675000000000002e-05 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.467528+0000", 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029457067 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.299 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.6 68/66 [PushOp(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head, version: 63'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head@63'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 86.812581350000002, 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.061983475000000003, 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.300 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.472013+0000", 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473325+0000", 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001312122 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473340+0000", 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5481e-05 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.473352+0000", 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.136e-05 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502239+0000", 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028887257999999999 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.502249+0000", 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.046e-05 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:03.533997+0000", 2024-09-21T15:18:30.303 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031747565999999998 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:98 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 0~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.115446113999994, 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.046016557999999999, 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.304 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 98 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169149+0000", 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.305 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169147+0000", 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169527+0000", 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038022899999999999 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169558+0000", 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0836000000000001e-05 2024-09-21T15:18:30.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169573+0000", 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5339999999999999e-05 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169736+0000", 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000162177 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169976+0000", 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000240671 2024-09-21T15:18:30.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170472+0000", 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00049572699999999995 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203487+0000", 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033014689999999999 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215061+0000", 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011574542 2024-09-21T15:18:30.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215120+0000", 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9020000000000001e-05 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215165+0000", 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.5118999999999999e-05 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:99 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.114996136000002, 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045741247999999998, 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 99 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.310 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169599+0000", 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169598+0000", 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169811+0000", 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021347299999999999 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169824+0000", 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2566e-05 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169833+0000", 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9369999999999993e-06 2024-09-21T15:18:30.312 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181062+0000", 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011229548000000001 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181172+0000", 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000109697 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.181308+0000", 2024-09-21T15:18:30.313 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000136124 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203492+0000", 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022183749999999999 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215251+0000", 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011759072000000001 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.314 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215303+0000", 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.2417999999999999e-05 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215340+0000", 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6600000000000002e-05 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:100 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.315 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.114746271000001, 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045671805000000003, 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 100 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:30.316 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169849+0000", 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.169848+0000", 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.317 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170047+0000", 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000199519 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170087+0000", 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9823000000000001e-05 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170114+0000", 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.7512999999999999e-05 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184787+0000", 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014672621 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.184940+0000", 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015298699999999999 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.185179+0000", 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023939299999999999 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.214997+0000", 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.02981787 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215437+0000", 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00043946799999999999 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215480+0000", 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.3516000000000001e-05 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215520+0000", 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.0024999999999997e-05 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:101 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.114470767, 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045543497000000002, 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.321 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 101 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170124+0000", 2024-09-21T15:18:30.322 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170123+0000", 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170331+0000", 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000207939 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.323 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170346+0000", 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.525e-05 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170355+0000", 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.4600000000000003e-06 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192764+0000", 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022408898 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.324 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.192918+0000", 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00015443299999999999 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.193090+0000", 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000171557 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215004+0000", 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021913986 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.325 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215581+0000", 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00057763200000000002 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215632+0000", 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.0253000000000001e-05 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215668+0000", 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5882e-05 2024-09-21T15:18:30.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:102 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.114067801999994, 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045262785999999999, 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.327 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 102 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170527+0000", 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.328 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170526+0000", 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170707+0000", 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018070299999999999 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170718+0000", 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1406000000000001e-05 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170726+0000", 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4200000000000001e-06 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.197936+0000", 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027210450000000001 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198014+0000", 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7776999999999998e-05 2024-09-21T15:18:30.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.198133+0000", 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011943 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215362+0000", 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017228841000000002 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.331 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215715+0000", 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00035230999999999999 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215748+0000", 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3801999999999999e-05 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215790+0000", 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1412999999999998e-05 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:103 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.113861765999999, 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045183032999999997, 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 103 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170733+0000", 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170732+0000", 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:30.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170913+0000", 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000180893 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170923+0000", 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6630000000000005e-06 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170930+0000", 2024-09-21T15:18:30.335 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2169999999999997e-06 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203556+0000", 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032625941999999998 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203643+0000", 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7256000000000006e-05 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.203768+0000", 2024-09-21T15:18:30.336 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012433999999999999 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215371+0000", 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011603474000000001 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215838+0000", 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00046702700000000001 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215881+0000", 2024-09-21T15:18:30.337 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2969999999999997e-05 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215916+0000", 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4961e-05 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:104 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.113657763000006, 2024-09-21T15:18:30.338 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045115492, 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 104 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170937+0000", 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.170936+0000", 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171099+0000", 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016227400000000001 2024-09-21T15:18:30.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171114+0000", 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5006e-05 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171121+0000", 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2729999999999997e-06 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209085+0000", 2024-09-21T15:18:30.341 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.037963964000000003 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209185+0000", 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9862e-05 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.209309+0000", 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000124695 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.342 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215929+0000", 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066199869999999999 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.215985+0000", 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5611e-05 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216022+0000", 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7469999999999999e-05 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.343 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216053+0000", 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0066999999999999e-05 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:105 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 74.113466880999994, 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.051278327999999998, 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.344 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 105 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171128+0000", 2024-09-21T15:18:30.345 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171127+0000", 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171283+0000", 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000156007 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.346 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171313+0000", 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9832000000000002e-05 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.171331+0000", 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8162999999999999e-05 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216082+0000", 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.044750834000000003 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.347 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216166+0000", 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3672999999999997e-05 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.216280+0000", 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000114234 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.221832+0000", 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0055522710000000001 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.348 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222313+0000", 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00048044400000000001 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222373+0000", 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0328999999999998e-05 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:16.222406+0000", 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3337000000000003e-05 2024-09-21T15:18:30.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:200 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 262144~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.976403555000005, 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053098463999999998, 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.350 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 200 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308191+0000", 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.351 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308190+0000", 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308291+0000", 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100892 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308384+0000", 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2437999999999998e-05 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308415+0000", 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1081999999999998e-05 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322831+0000", 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014416725 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322883+0000", 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1122999999999998e-05 2024-09-21T15:18:30.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.322962+0000", 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9098999999999996e-05 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338265+0000", 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015303740999999999 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361056+0000", 2024-09-21T15:18:30.354 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022790431 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361203+0000", 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014696300000000001 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361290+0000", 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6929999999999996e-05 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:201 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 393216~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.976168533000006, 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.053175820999999998, 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 201 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308426+0000", 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308425+0000", 2024-09-21T15:18:30.357 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308591+0000", 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016595999999999999 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308609+0000", 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8113000000000001e-05 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.358 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308618+0000", 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.7930000000000003e-06 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326446+0000", 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.017827867000000001 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326493+0000", 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.7457999999999999e-05 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.359 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.326566+0000", 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2157999999999996e-05 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345729+0000", 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.019163139999999999 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361375+0000", 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015646277 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361469+0000", 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3961000000000004e-05 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.361602+0000", 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133084 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:202 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 524288~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:30.361 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.975968109999997, 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058377694000000001, 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 202 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.362 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308627+0000", 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308626+0000", 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.363 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308929+0000", 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00030262799999999999 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308967+0000", 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8782000000000002e-05 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.308984+0000", 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6753e-05 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.364 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330071+0000", 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021086546000000001 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330194+0000", 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000123182 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.330376+0000", 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018249299999999999 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.365 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345732+0000", 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.015355671 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366849+0000", 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021116956999999999 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.366942+0000", 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.3276000000000006e-05 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367004+0000", 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2182999999999997e-05 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:203 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 655360~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.975564528000007, 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058300422999999997, 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.367 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 203 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309030+0000", 2024-09-21T15:18:30.368 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309029+0000", 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309248+0000", 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021858899999999999 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.369 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309266+0000", 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8176e-05 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309280+0000", 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4377e-05 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334582+0000", 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025301889000000001 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.370 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334633+0000", 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1177000000000001e-05 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.334711+0000", 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.7558999999999994e-05 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345736+0000", 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011025563 2024-09-21T15:18:30.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367118+0000", 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021381714 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367236+0000", 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000117812 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367331+0000", 2024-09-21T15:18:30.372 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4864000000000005e-05 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:204 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 786432~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.975305371000005, 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.058272110000000002, 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.373 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 204 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309289+0000", 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309462+0000", 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00017298500000000001 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309472+0000", 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0637e-05 2024-09-21T15:18:30.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309479+0000", 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.579e-06 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338296+0000", 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028817427 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338347+0000", 2024-09-21T15:18:30.376 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1223000000000001e-05 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.338422+0000", 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.4964999999999998e-05 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.345739+0000", 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007316131 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.377 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367442+0000", 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.021703792999999999 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367523+0000", 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0578e-05 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.367562+0000", 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.8612000000000002e-05 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.378 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:205 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 917504~131072] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.975110143999999, 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.059855925999999997, 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 205 2024-09-21T15:18:30.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309485+0000", 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309484+0000", 2024-09-21T15:18:30.380 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309607+0000", 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012254599999999999 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309651+0000", 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.381 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.309658+0000", 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9299999999999997e-06 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342078+0000", 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.032420005000000002 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342137+0000", 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8797e-05 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.382 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.342214+0000", 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.6755999999999995e-05 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.346405+0000", 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041910619999999997 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369141+0000", 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.022736290999999999 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.383 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369231+0000", 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9758000000000004e-05 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.369341+0000", 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010992099999999999 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5557.0:286 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 131072~917504] snapc 6=[6,4] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T15:18:30.384 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 72.780858379999998, 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.040449859999999997, 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5557", 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/1520933397", 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 286 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:30.385 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503736+0000", 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.503734+0000", 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.386 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504548+0000", 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081411000000000005 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504582+0000", 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3386000000000002e-05 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504602+0000", 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.0146e-05 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.387 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504715+0000", 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011267100000000001 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.504936+0000", 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022115000000000001 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.505260+0000", 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000323985 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.388 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.536419+0000", 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031159229 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544065+0000", 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.007645911 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544135+0000", 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9927999999999999e-05 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.389 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:17:17.544186+0000", 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1400999999999998e-05 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "MOSDPGPush(2.7 116/115 [PushOp(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head, version: 50'24, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ef1c8a18:ns::rbd_data.137aae78387e.0000000000000003:head@50'24, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:18:29.236968+0000", 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 1.047627171, 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.047763545999999997, 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.390 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.236968+0000", 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.236968+0000", 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.236966+0000", 2024-09-21T15:18:30.391 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238821+0000", 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001854957 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238844+0000", 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.2498e-05 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.392 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238857+0000", 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3287e-05 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238974+0000", 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000116589 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.238995+0000", 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.1311e-05 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.393 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:18:29.284731+0000", 2024-09-21T15:18:30.394 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.045736227999999997 2024-09-21T15:18:30.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.394 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.394 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.394 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:30.395 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:30.302+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:30.396 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:30.396 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:30.394+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:30.496 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:30.495+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:30.511 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:30.511 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:30.511 INFO:teuthology.orchestra.run.smithi142.stdout: "num_ops": 0 2024-09-21T15:18:30.512 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:30.521 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:30.597 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:30.598+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:30.624 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:30.624 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [], 2024-09-21T15:18:30.624 INFO:teuthology.orchestra.run.smithi142.stdout: "complaint_time": 30, 2024-09-21T15:18:30.624 INFO:teuthology.orchestra.run.smithi142.stdout: "num_blocked_ops": 0 2024-09-21T15:18:30.624 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:30.634 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:30.698 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:30.695+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:30.746 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "size": 20, 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 600, 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "ops": [ 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:344 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 109.64086370299999, 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089745640000000008, 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.747 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 344 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102600+0000", 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.748 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102599+0000", 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102603+0000", 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7440000000000001e-06 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102615+0000", 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.183e-05 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102640+0000", 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5743000000000001e-05 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110752+0000", 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0081118609999999997 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.110939+0000", 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018720600000000001 2024-09-21T15:18:30.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111574+0000", 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063478700000000005 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:345 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 109.64081265999999, 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0098217900000000004, 2024-09-21T15:18:30.751 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 345 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.752 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102651+0000", 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102650+0000", 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102654+0000", 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7069999999999998e-06 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.753 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102665+0000", 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.102672+0000", 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9330000000000004e-06 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111617+0000", 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089454110000000003 2024-09-21T15:18:30.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.111738+0000", 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000120326 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.112472+0000", 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00073462600000000005 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.755 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:360 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 109.62519311699999, 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083178980000000007, 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 360 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:30.756 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118270+0000", 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118273+0000", 2024-09-21T15:18:30.757 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1970000000000001e-06 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118284+0000", 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1206000000000001e-05 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118290+0000", 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9240000000000004e-06 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125859+0000", 2024-09-21T15:18:30.758 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0075693949999999996 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.125951+0000", 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.1675000000000006e-05 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126588+0000", 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00063711799999999999 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:361 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55)", 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 109.62516636399999, 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0090096699999999991, 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "started", 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 361 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.760 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118297+0000", 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118296+0000", 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.761 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118299+0000", 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.14e-06 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118310+0000", 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0973e-05 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.118317+0000", 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.207e-06 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.762 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126654+0000", 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0083377850000000003 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.126749+0000", 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4847999999999997e-05 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:41.127307+0000", 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00055732999999999998 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.763 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:665 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.456118261, 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.014606948, 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 665 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287345+0000", 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287342+0000", 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999971 2024-09-21T15:18:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287756+0000", 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00041337500000000001 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287789+0000", 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.3657e-05 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287806+0000", 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7220000000000001e-05 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287956+0000", 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014966600000000001 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.287974+0000", 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.7906000000000001e-05 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292095+0000", 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0041210960000000003 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292318+0000", 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00022283599999999999 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.292700+0000", 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00038199600000000001 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301687+0000", 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0089867860000000001 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301891+0000", 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000204799 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301926+0000", 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.5012999999999998e-05 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.301952+0000", 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5474e-05 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:666 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.440973404, 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027744641, 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 666 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302490+0000", 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.302488+0000", 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303428+0000", 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00093968899999999995 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303446+0000", 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8202999999999998e-05 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303455+0000", 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303497+0000", 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1972999999999999e-05 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303556+0000", 2024-09-21T15:18:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9304e-05 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.303639+0000", 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3157000000000004e-05 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.327255+0000", 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023615678000000001 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330135+0000", 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0028803040000000002 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330192+0000", 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6716999999999999e-05 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.330235+0000", 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2679000000000003e-05 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:667 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.411826588, 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088238220000000003, 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 667 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331637+0000", 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331631+0000", 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999948 2024-09-21T15:18:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331732+0000", 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100142 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331752+0000", 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9958000000000001e-05 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331765+0000", 2024-09-21T15:18:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.3030000000000001e-05 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.331851+0000", 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6844999999999999e-05 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332024+0000", 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000172373 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.332266+0000", 2024-09-21T15:18:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000242286 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.339683+0000", 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0074166730000000004 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340330+0000", 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064767999999999998 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340407+0000", 2024-09-21T15:18:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.674e-05 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340461+0000", 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.3307000000000003e-05 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:668 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.402499683, 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026314581, 2024-09-21T15:18:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 668 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340964+0000", 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.340962+0000", 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341782+0000", 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00081934399999999995 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341797+0000", 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4883e-05 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341803+0000", 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341858+0000", 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4849999999999998e-05 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.341937+0000", 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9160999999999996e-05 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.342032+0000", 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.4753000000000006e-05 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.365341+0000", 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023309248000000001 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367135+0000", 2024-09-21T15:18:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0017939290000000001 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367218+0000", 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.2717000000000001e-05 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.367278+0000", 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0346000000000001e-05 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:669 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.374863531, 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010513005000000001, 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 669 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368600+0000", 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368599+0000", 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368708+0000", 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00010957 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368720+0000", 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.218e-05 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368726+0000", 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.6919999999999999e-06 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368784+0000", 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.8446000000000003e-05 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.368929+0000", 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000144556 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.369136+0000", 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00020665700000000001 2024-09-21T15:18:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378314+0000", 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0091782819999999994 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.378964+0000", 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00064981999999999995 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379051+0000", 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.6744999999999997e-05 2024-09-21T15:18:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379113+0000", 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2236999999999994e-05 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:670 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.363785826, 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028306417, 2024-09-21T15:18:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 670 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379677+0000", 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.379676+0000", 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380469+0000", 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00079255799999999996 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380484+0000", 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5624000000000002e-05 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380497+0000", 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.2248e-05 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380630+0000", 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133788 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380765+0000", 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000134208 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.380975+0000", 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00021051899999999999 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407092+0000", 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026116650000000002 2024-09-21T15:18:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407927+0000", 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083530099999999999 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407959+0000", 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1946999999999998e-05 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.407984+0000", 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4776000000000001e-05 2024-09-21T15:18:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:672 2.4 2:2e35f377:::rbd_data.13743413a48b.0000000000000008:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 106.329016279, 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.033932087999999999, 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 672 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414447+0000", 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.414446+0000", 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415297+0000", 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085151499999999995 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415317+0000", 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9553000000000001e-05 2024-09-21T15:18:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415327+0000", 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.0042e-05 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415364+0000", 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.7344e-05 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415410+0000", 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.6306e-05 2024-09-21T15:18:30.801 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.415482+0000", 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1891999999999995e-05 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.439465+0000", 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.023982781000000002 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.802 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448304+0000", 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0088392680000000008 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448348+0000", 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.4020999999999997e-05 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:44.448379+0000", 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.0670999999999998e-05 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.803 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:713 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.440492197, 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0096771879999999998, 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 713 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.804 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302971+0000", 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.302968+0000", 2024-09-21T15:18:30.805 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999967 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303263+0000", 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00029564899999999999 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303300+0000", 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.6480000000000003e-05 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.806 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303316+0000", 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.5815999999999999e-05 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303441+0000", 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012505099999999999 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303539+0000", 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.8596000000000003e-05 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.807 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.303675+0000", 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013546899999999999 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.309593+0000", 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0059182849999999997 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312511+0000", 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0029181649999999999 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312594+0000", 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.3009000000000004e-05 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.312648+0000", 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.4117999999999999e-05 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:714 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:30.809 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.43023712999999, 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.030601535999999999, 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 714 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.810 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313226+0000", 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.313225+0000", 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314323+0000", 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0010980670000000001 2024-09-21T15:18:30.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314348+0000", 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4994000000000002e-05 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314368+0000", 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9896999999999999e-05 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314460+0000", 2024-09-21T15:18:30.812 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.2692e-05 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314622+0000", 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00016186600000000001 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.314857+0000", 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00023476200000000001 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.813 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343380+0000", 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028522631999999999 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343721+0000", 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00034150499999999998 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343772+0000", 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.1181999999999999e-05 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.814 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.343828+0000", 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.5408999999999998e-05 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:715 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.398190133, 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.011597847, 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.815 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 715 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345273+0000", 2024-09-21T15:18:30.816 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345271+0000", 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345453+0000", 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000181986 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345481+0000", 2024-09-21T15:18:30.817 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.8413000000000001e-05 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345501+0000", 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9653e-05 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345550+0000", 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8977999999999998e-05 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.818 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345666+0000", 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011584899999999999 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.345816+0000", 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00014997099999999999 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.350180+0000", 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0043647299999999998 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.819 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356815+0000", 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0066344589999999997 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356846+0000", 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.1170000000000001e-05 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.356871+0000", 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4915000000000001e-05 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.820 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:716 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.38597818, 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.027134019999999998, 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 716 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.821 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357485+0000", 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.357484+0000", 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.822 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358310+0000", 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082578600000000001 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358330+0000", 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358339+0000", 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.9639999999999992e-06 2024-09-21T15:18:30.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358406+0000", 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.7720999999999998e-05 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358495+0000", 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.8245000000000006e-05 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.358623+0000", 2024-09-21T15:18:30.824 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00012852999999999999 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.383631+0000", 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025007913999999999 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384512+0000", 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00088115600000000002 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.825 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384547+0000", 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.4635000000000003e-05 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.384619+0000", 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.2175999999999995e-05 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:718 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:30.826 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.35208222200001, 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.031219971999999999, 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 718 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391381+0000", 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.391380+0000", 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392207+0000", 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00082671700000000003 2024-09-21T15:18:30.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392219+0000", 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.1979e-05 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392225+0000", 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 5.9370000000000001e-06 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392294+0000", 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.9377999999999999e-05 2024-09-21T15:18:30.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392412+0000", 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00011777000000000001 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.392579+0000", 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000166677 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.421587+0000", 2024-09-21T15:18:30.830 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.029007807 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422422+0000", 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00083556600000000004 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422489+0000", 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.6600000000000006e-05 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.422601+0000", 2024-09-21T15:18:30.831 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000112215 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.4996.0:720 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 105.313037353, 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028704005000000001, 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.4996", 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:30.832 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 720 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430426+0000", 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.833 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.430425+0000", 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431286+0000", 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00086088600000000005 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431305+0000", 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.9587999999999999e-05 2024-09-21T15:18:30.834 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431315+0000", 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.9299999999999998e-06 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431378+0000", 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.2545999999999994e-05 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.835 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431478+0000", 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000100083 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.431612+0000", 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000133764 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.458171+0000", 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.026559117 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.836 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459030+0000", 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00085901100000000002 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459069+0000", 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 3.9403000000000002e-05 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:45.459130+0000", 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.0829999999999999e-05 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.837 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:164 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2097152~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.452594226000002, 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.010949963, 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 164 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.838 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290869+0000", 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290867+0000", 2024-09-21T15:18:30.839 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290963+0000", 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6352000000000002e-05 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.290987+0000", 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.4015999999999998e-05 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291001+0000", 2024-09-21T15:18:30.840 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.4070000000000001e-05 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291080+0000", 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.9097999999999994e-05 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op must wait for map", 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.291088+0000", 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.8970000000000008e-06 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.841 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.293910+0000", 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.002822462 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294037+0000", 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000126234 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.294226+0000", 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00018944799999999999 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.842 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301139+0000", 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.0069132639999999997 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301705+0000", 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00056563499999999999 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301777+0000", 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 7.1443e-05 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.843 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.301819+0000", 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.2499000000000001e-05 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:165 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.441016719000004, 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.028234764999999998, 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.844 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 165 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302447+0000", 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.302445+0000", 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.999999 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303193+0000", 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00074781700000000001 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303209+0000", 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.6002999999999999e-05 2024-09-21T15:18:30.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303219+0000", 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 9.6809999999999993e-06 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303326+0000", 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000107192 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303465+0000", 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00013921300000000001 2024-09-21T15:18:30.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.303705+0000", 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000239647 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.328750+0000", 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.025045392 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330626+0000", 2024-09-21T15:18:30.848 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.001875247 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330655+0000", 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.9040999999999999e-05 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.330681+0000", 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.6673e-05 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.849 INFO:teuthology.orchestra.run.smithi142.stdout: "description": "osd_op(client.5206.0:167 2.1 2:9c925b24:::rbd_data.13743413a48b.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "initiated_at": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "age": 93.406605951000003, 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.038058690999999999, 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "type_data": { 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "client_info": { 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "client": "client.5206", 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "tid": 167 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "events": [ 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "initiated", 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:30.850 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "throttled", 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336857+0000", 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "header_read", 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.336856+0000", 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "all_read", 2024-09-21T15:18:30.851 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337763+0000", 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.00090762799999999995 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "dispatched", 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337789+0000", 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 2.5239999999999999e-05 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "queued_for_pg", 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337807+0000", 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 1.8712999999999999e-05 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "reached_pg", 2024-09-21T15:18:30.852 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337856+0000", 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.8226999999999998e-05 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "started", 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337905+0000", 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.9712000000000001e-05 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.337986+0000", 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 8.0346000000000006e-05 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.853 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "op_commit", 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374382+0000", 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.036396378 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374811+0000", 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 0.000429047 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "commit_sent", 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374874+0000", 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 6.3236000000000002e-05 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-21T15:18:30.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: "event": "done", 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: "time": "2024-09-21T15:16:57.374916+0000", 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: "duration": 4.1677000000000001e-05 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-21T15:18:30.855 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-21T15:18:30.856 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:30.855+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:30.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:18:30.958 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:30.958+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:30.974 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:30.975 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:30.975 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-09-21T15:18:30.975 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:30.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:18:31.058 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:31.058+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:31.090 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-09-21T15:18:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-09-21T15:18:31.090 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-09-21T15:18:31.090 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:31.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:18:31.159 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:31.155+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:31.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:18:31.215 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:31.215 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-09-21T15:18:31.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-09-21T15:18:31.215 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-09-21T15:18:31.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.215 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:626 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T15:18:31.215 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.755904055, 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040376593000000002, 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 626 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457810+0000", 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.457809+0000", 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458808+0000", 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099966599999999997 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458839+0000", 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0571000000000003e-05 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458853+0000", 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3925e-05 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458895+0000", 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:18:31.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.458960+0000", 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5294000000000002e-05 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.459048+0000", 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7504999999999997e-05 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.497375+0000", 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038327716999999997 2024-09-21T15:18:31.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498081+0000", 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070553100000000002 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498142+0000", 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0986000000000002e-05 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:43.498187+0000", 2024-09-21T15:18:31.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5176000000000003e-05 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:676 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.708019798, 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039905346000000001, 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.221 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505695+0000", 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.505693+0000", 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506551+0000", 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085803500000000005 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506569+0000", 2024-09-21T15:18:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8074999999999999e-05 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506578+0000", 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5930000000000006e-06 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506630+0000", 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1515999999999999e-05 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506719+0000", 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9000999999999997e-05 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.506810+0000", 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1920000000000001e-05 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.531039+0000", 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024228855000000001 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545482+0000", 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014442692 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545546+0000", 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4312000000000004e-05 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.545600+0000", 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3714000000000001e-05 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.226 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:678 2.2 2:4ff9fbb4:::rbd_data.13743413a48b.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.65984795200001, 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042214693999999997, 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 678 2024-09-21T15:18:31.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553867+0000", 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.553864+0000", 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999976 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554712+0000", 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084738100000000002 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554744+0000", 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2772999999999999e-05 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554764+0000", 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9488999999999999e-05 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554831+0000", 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6767000000000006e-05 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.554943+0000", 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011231900000000001 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.555116+0000", 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017303899999999999 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.593274+0000", 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038157486999999997 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.595947+0000", 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026736080000000001 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596035+0000", 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7964e-05 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.596081+0000", 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106000000000002e-05 2024-09-21T15:18:31.232 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:696 2.7 2:f1f9bd74:::rbd_data.13743413a48b.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.30167885, 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038358727000000002, 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:31.233 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 696 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912036+0000", 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912035+0000", 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912863+0000", 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082834900000000003 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912876+0000", 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3081000000000001e-05 2024-09-21T15:18:31.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912882+0000", 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912926+0000", 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4218999999999998e-05 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.912980+0000", 2024-09-21T15:18:31.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3677000000000001e-05 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.913054+0000", 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4683000000000003e-05 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.948740+0000", 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035685770999999998 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950334+0000", 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015943089999999999 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950367+0000", 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2419000000000003e-05 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:44.950394+0000", 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7630000000000001e-05 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:704 2.6 2:6f358a07:::rbd_data.13743413a48b.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.13569497899999, 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042282467999999997, 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 704 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078020+0000", 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078017+0000", 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:31.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078843+0000", 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000825131 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078873+0000", 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0605999999999998e-05 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078896+0000", 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2466000000000001e-05 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.078960+0000", 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4461000000000005e-05 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079055+0000", 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5147000000000002e-05 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.079198+0000", 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014274099999999999 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.119654+0000", 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040456199999999998 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120220+0000", 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056561100000000004 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120273+0000", 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3125999999999999e-05 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.120302+0000", 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9085e-05 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4996.0:712 2.6 2:68fde513:::rbd_data.13743413a48b.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 105.95018971, 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039385278000000003, 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4996", 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3389374322", 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 712 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263525+0000", 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.263523+0000", 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:31.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264380+0000", 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00085712799999999997 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264405+0000", 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367999999999998e-05 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264420+0000", 2024-09-21T15:18:31.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4793999999999999e-05 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264505+0000", 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4724999999999999e-05 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264573+0000", 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7648e-05 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:31.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.264666+0000", 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3591000000000006e-05 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.301773+0000", 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037107306 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302803+0000", 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010293069999999999 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302863+0000", 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0307999999999999e-05 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:45.302910+0000", 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6998e-05 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:23 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:31.250 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.971550879000006, 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037218474000000001, 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 23 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242164+0000", 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242162+0000", 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242804+0000", 2024-09-21T15:18:31.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064144899999999999 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242843+0000", 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9569e-05 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242876+0000", 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2373000000000003e-05 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253622+0000", 2024-09-21T15:18:31.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010746217000000001 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253721+0000", 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9079999999999993e-05 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.253890+0000", 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169233 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274049+0000", 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020158697 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279303+0000", 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052537160000000003 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279338+0000", 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5592000000000001e-05 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279382+0000", 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3701000000000001e-05 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:24 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.970766002000005, 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036646094999999997, 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 24 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242948+0000", 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.242947+0000", 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243194+0000", 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024651399999999998 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243217+0000", 2024-09-21T15:18:31.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3697e-05 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243234+0000", 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6769000000000002e-05 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259692+0000", 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016458181999999998 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259762+0000", 2024-09-21T15:18:31.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9330000000000002e-05 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.259869+0000", 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107268 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279400+0000", 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019530585 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279529+0000", 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012973699999999999 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279558+0000", 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8608000000000001e-05 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279595+0000", 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6619000000000003e-05 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:26 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.970225877999994, 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620206000000003, 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 26 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.262 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243489+0000", 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243488+0000", 2024-09-21T15:18:31.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243659+0000", 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017080900000000001 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243676+0000", 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7306999999999999e-05 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243684+0000", 2024-09-21T15:18:31.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274085+0000", 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.03040116 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274194+0000", 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108762 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.274350+0000", 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015661900000000001 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291810+0000", 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017459453 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295906+0000", 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004096545 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.295955+0000", 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8571e-05 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296109+0000", 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153907 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:27 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:31.267 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.970023462, 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052618531000000003, 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 27 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:31.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243691+0000", 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243690+0000", 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243855+0000", 2024-09-21T15:18:31.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016495 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243872+0000", 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6776e-05 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243880+0000", 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7360000000000001e-06 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279778+0000", 2024-09-21T15:18:31.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035897749 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.279900+0000", 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122063 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.280038+0000", 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013876399999999999 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296138+0000", 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016099445 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296188+0000", 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0693999999999999e-05 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296274+0000", 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5972999999999998e-05 2024-09-21T15:18:31.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296310+0000", 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5120999999999998e-05 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:28 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.969827257000006, 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052620759000000003, 2024-09-21T15:18:31.273 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 28 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243887+0000", 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.243886+0000", 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244050+0000", 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016375299999999999 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244066+0000", 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5913000000000002e-05 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244074+0000", 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7330000000000003e-06 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285845+0000", 2024-09-21T15:18:31.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041770801000000003 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.285957+0000", 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112075 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.286099+0000", 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014275600000000001 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296414+0000", 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010314464000000001 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296427+0000", 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.327e-05 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296472+0000", 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5136999999999998e-05 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296508+0000", 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5630000000000003e-05 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:29 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.969633134000006, 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052556139000000002, 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 29 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244081+0000", 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244080+0000", 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.999999 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244238+0000", 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157403 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244254+0000", 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6166000000000001e-05 2024-09-21T15:18:31.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.244262+0000", 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2230000000000007e-06 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291867+0000", 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047604654000000003 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.291992+0000", 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012561699999999999 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.292161+0000", 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016820999999999999 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296552+0000", 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043911920000000004 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296565+0000", 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3426e-05 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296609+0000", 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3837000000000001e-05 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.296637+0000", 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354e-05 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.284 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:34 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.915185913000002, 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047651179000000002, 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 34 2024-09-21T15:18:31.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298529+0000", 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298528+0000", 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298618+0000", 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0208000000000001e-05 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298629+0000", 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1026000000000001e-05 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298634+0000", 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.13e-06 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327412+0000", 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028777805 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327504+0000", 2024-09-21T15:18:31.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2111e-05 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.327647+0000", 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014291600000000001 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.345995+0000", 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018347413 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346015+0000", 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9885999999999999e-05 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346105+0000", 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9920000000000006e-05 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346180+0000", 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5247999999999995e-05 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:35 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.915053466000003, 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047870083000000001, 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 35 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.291 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298661+0000", 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:31.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298789+0000", 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127988 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298800+0000", 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1922999999999999e-05 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298807+0000", 2024-09-21T15:18:31.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799999999999998e-06 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331336+0000", 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032528702999999999 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331495+0000", 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015899499999999999 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.331728+0000", 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023354100000000001 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346197+0000", 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014469044 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346333+0000", 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136186 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346456+0000", 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122564 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346531+0000", 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5145e-05 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:36 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.296 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.914902087000002, 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047963459, 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 36 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298812+0000", 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298920+0000", 2024-09-21T15:18:31.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107792 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298930+0000", 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.058e-05 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298935+0000", 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7929999999999997e-06 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337159+0000", 2024-09-21T15:18:31.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038224414999999998 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337316+0000", 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015642100000000001 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.337598+0000", 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028205899999999998 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346628+0000", 2024-09-21T15:18:31.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090296649999999992 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346642+0000", 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4090000000000001e-05 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346719+0000", 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6845e-05 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346776+0000", 2024-09-21T15:18:31.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7290999999999997e-05 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:37 2.5 2:b723daed:::rbd_data.13743413a48b.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.914774715999997, 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049670370999999998, 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 37 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298940+0000", 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.298939+0000", 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999995 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299038+0000", 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8883999999999997e-05 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299048+0000", 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5440000000000004e-06 2024-09-21T15:18:31.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.299053+0000", 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0860000000000001e-06 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342252+0000", 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043199277000000001 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342345+0000", 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3000999999999999e-05 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.342498+0000", 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015331500000000001 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.346794+0000", 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042953569999999996 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348540+0000", 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001745955 2024-09-21T15:18:31.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348568+0000", 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.817e-05 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.348610+0000", 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2276e-05 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:44 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.779974715999998, 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044808293999999999, 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433740+0000", 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.433738+0000", 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999981 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434723+0000", 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098563899999999996 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434755+0000", 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1492000000000003e-05 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434773+0000", 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7935999999999999e-05 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.434876+0000", 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103305 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435037+0000", 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016103299999999999 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.435283+0000", 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024623400000000001 2024-09-21T15:18:31.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.477507+0000", 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042223241000000002 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478469+0000", 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00096199199999999995 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478509+0000", 2024-09-21T15:18:31.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0447000000000001e-05 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.478548+0000", 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8970999999999997e-05 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:50 2.6 2:6473f039:::rbd_data.13743413a48b.0000000000000001:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:31.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.642948632, 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042378401000000003, 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570766+0000", 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.570764+0000", 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571601+0000", 2024-09-21T15:18:31.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00083664399999999999 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571626+0000", 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5054000000000001e-05 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571638+0000", 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1688000000000001e-05 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571694+0000", 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6070999999999999e-05 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571784+0000", 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9864000000000006e-05 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.571921+0000", 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013765899999999999 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.611897+0000", 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039975840999999998 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613017+0000", 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011203750000000001 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613081+0000", 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3257999999999996e-05 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.613144+0000", 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3532000000000001e-05 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:59 2.7 2:f371c287:::rbd_data.13743413a48b.0000000000000002:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.467651728000007, 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039111193000000002, 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 59 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746063+0000", 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746061+0000", 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746899+0000", 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000837319 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746913+0000", 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4651000000000001e-05 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746921+0000", 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.746968+0000", 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7346999999999999e-05 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747047+0000", 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8306999999999994e-05 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.747155+0000", 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108422 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.769868+0000", 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022713110000000002 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785106+0000", 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015237858999999999 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785148+0000", 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1773000000000001e-05 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.785174+0000", 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6279999999999999e-05 2024-09-21T15:18:31.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.5206.0:62 2.7 2:eb46bd6e:::rbd_data.13743413a48b.0000000000000003:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.417359203000004, 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037341043999999997, 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.5206", 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "172.21.15.146:0/3470968364", 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 62 2024-09-21T15:18:31.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796355+0000", 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-09-21T15:18:31.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.796354+0000", 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4294967295.9999986 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797224+0000", 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00087020000000000001 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797241+0000", 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7621e-05 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-09-21T15:18:31.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797250+0000", 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0070000000000003e-06 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797312+0000", 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1371000000000006e-05 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797438+0000", 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012574400000000001 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-09-21T15:18:31.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.797619+0000", 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018168399999999999 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.830366+0000", 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032746863000000001 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833562+0000", 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003196263 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-09-21T15:18:31.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833637+0000", 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4771000000000006e-05 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-09-21T15:16:54.833696+0000", 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9169999999999998e-05 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-21T15:18:31.330 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:31.332 INFO:tasks.workunit.client.0.smithi146.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T15:18:31.332 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd17425 2024-09-21T15:18:31.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:18:31.333 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:18:31.334 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:31.327+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:31.343 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-09-21T15:18:31.367 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-09-21T15:18:31.404 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-09-21T15:18:31.432 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-09-21T15:18:31.430+0000 7fe6874f3700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21000) UID: 0 2024-09-21T15:18:31.442 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-09-21T15:18:31.467 INFO:tasks.workunit.client.0.smithi146.stderr:2024-09-21T15:18:31.465+0000 7f4afbe70700 0 -- 172.21.15.146:0/1674680650 >> [v2:172.21.15.142:6816/3606282773,v1:172.21.15.142:6817/3606282773] conn(0x560d2c1dbdf0 msgr2=0x560d2c1de1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:18:31.477 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-09-21T15:18:31.487 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:31.487 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:31.499 INFO:tasks.workunit.client.0.smithi146.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-21T15:18:31.497+0000 7f4afbe70700 0 -- 172.21.15.146:0/1674680650 >> [v2:172.21.15.142:6816/3606282773,v1:172.21.15.142:6817/3606282773] conn(0x7f4ae003afd0 msgr2=0x7f4ae003d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T15:18:31.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:18:31.510 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-09-21T15:18:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:18:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:18:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:18:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:18:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:18:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:18:31.524 INFO:tasks.workunit.client.0.smithi146.stderr: Removing image: 100% complete...done. 2024-09-21T15:18:31.532 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd namespace remove rbd/ns 2024-09-21T15:18:31.533 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:31.531+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:31.611 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:31.611 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:31.615 INFO:tasks.workunit.client.0.smithi146.stderr:+ rbd config global rm global rbd_default_pool 2024-09-21T15:18:31.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:18:31.634 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:31.631+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:31.666 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph osd pool delete new_default_pool17425 new_default_pool17425 --yes-i-really-really-mean-it 2024-09-21T15:18:31.728 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:31.728 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:31.735 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:31.731+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:31.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:18:31.836 INFO:tasks.ceph.osd.4.smithi142.stderr:2024-09-21T15:18:31.831+0000 7f926a099700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 21101) UID: 0 2024-09-21T15:18:31.847 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:31.848 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:31.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:18:31.937 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:31.935+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:31.968 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:31.968 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:31.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:18:32.037 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:32.035+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:32.085 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:32.085 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:32.095 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:18:32.140 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:32.139+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:32.212 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:32.212 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:32.222 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:18:32.238 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-09-21T15:18:32.238+0000 7fcffc5cf700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 21004) UID: 0 2024-09-21T15:18:32.244 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 2, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T15:18:32.244 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:18:32.244 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T15:18:32.244 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:18:32.244 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:18:32.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-21T15:18:32.332 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:32.332 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:32.340 INFO:tasks.ceph.osd.5.smithi142.stderr:2024-09-21T15:18:32.335+0000 7fa017a59700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 21103) UID: 0 2024-09-21T15:18:32.342 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:18:32.359 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:32.359 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T15:18:32.359 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:32.375 INFO:tasks.workunit.client.0.smithi146.stderr:pool 'new_default_pool17425' does not exist 2024-09-21T15:18:32.386 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T15:18:32.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T15:18:32.399 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-09-21T15:18:32.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T15:18:32.441 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:32.439+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:32.446 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:32.446 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:32.456 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:18:32.542 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-09-21T15:18:32.542+0000 7f5b869c8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 21003) UID: 0 2024-09-21T15:18:32.608 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:32.608 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:32.618 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:18:32.644 INFO:tasks.ceph.osd.3.smithi142.stderr:2024-09-21T15:18:32.639+0000 7fb97151e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 21099) UID: 0 2024-09-21T15:18:32.669 DEBUG:teuthology.parallel:result is None 2024-09-21T15:18:32.669 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T15:18:32.684 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T15:18:32.685 DEBUG:teuthology.orchestra.run.smithi146:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T15:18:32.693 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T15:18:32.694 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T15:18:32.706 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T15:18:32.728 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:32.728 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:32.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:18:32.846 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:32.846 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:32.856 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:18:32.967 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:33.016 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:33.018 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:18:33.129 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:33.129 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:33.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:18:33.247 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument 2024-09-21T15:18:33.248 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:18:33.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:18:34.228 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-09-21T15:18:34.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:18:35.292 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-09-21T15:18:37.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T15:18:37.517 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a: {} 2024-09-21T15:18:37.533 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b: {} 2024-09-21T15:18:37.547 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c: {} 2024-09-21T15:18:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:18:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:18:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:18:37.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T15:18:38.323 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T15:18:38.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-21T15:18:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T15:18:39.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T15:18:40.235 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T15:18:40.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-21T15:18:41.233 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T15:18:41.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T15:18:42.235 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T15:18:42.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-21T15:18:43.235 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T15:18:43.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T15:18:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T15:18:44.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-21T15:18:45.352 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T15:18:45.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T15:18:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T15:18:46.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-21T15:18:47.367 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T15:18:47.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T15:18:48.381 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T15:18:48.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-21T15:18:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T15:18:49.419 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T15:18:49.419 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T15:18:49.419 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T15:18:49.419 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T15:18:49.419 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T15:18:49.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:18:49.730 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:49.730 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":133,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:18:49.381564+0000","last_up_change":"2024-09-21T15:15:15.737314+0000","last_in_change":"2024-09-21T15:17:42.447038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:15:15.995966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:15:21.158896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6817","nonce":3600997419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6819","nonce":3600997419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6823","nonce":3600997419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6821","nonce":3600997419}]},"public_addr":"172.21.15.67:6817/3600997419","cluster_addr":"172.21.15.67:6819/3600997419","heartbeat_back_addr":"172.21.15.67:6823/3600997419","heartbeat_front_addr":"172.21.15.67:6821/3600997419","state":["exists","up"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6809","nonce":1817390160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6811","nonce":1817390160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6815","nonce":1817390160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6813","nonce":1817390160}]},"public_addr":"172.21.15.67:6809/1817390160","cluster_addr":"172.21.15.67:6811/1817390160","heartbeat_back_addr":"172.21.15.67:6815/1817390160","heartbeat_front_addr":"172.21.15.67:6813/1817390160","state":["exists","up"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6801","nonce":3371454484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6803","nonce":3371454484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6807","nonce":3371454484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6805","nonce":3371454484}]},"public_addr":"172.21.15.67:6801/3371454484","cluster_addr":"172.21.15.67:6803/3371454484","heartbeat_back_addr":"172.21.15.67:6807/3371454484","heartbeat_front_addr":"172.21.15.67:6805/3371454484","state":["exists","up"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6809","nonce":1069930087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6811","nonce":1069930087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6815","nonce":1069930087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6813","nonce":1069930087}]},"public_addr":"172.21.15.142:6809/1069930087","cluster_addr":"172.21.15.142:6811/1069930087","heartbeat_back_addr":"172.21.15.142:6815/1069930087","heartbeat_front_addr":"172.21.15.142:6813/1069930087","state":["exists","up"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6817","nonce":3606282773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6819","nonce":3606282773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6823","nonce":3606282773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6821","nonce":3606282773}]},"public_addr":"172.21.15.142:6817/3606282773","cluster_addr":"172.21.15.142:6819/3606282773","heartbeat_back_addr":"172.21.15.142:6823/3606282773","heartbeat_front_addr":"172.21.15.142:6821/3606282773","state":["exists","up"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6801","nonce":3327577487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6803","nonce":3327577487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6807","nonce":3327577487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6805","nonce":3327577487}]},"public_addr":"172.21.15.142:6801/3327577487","cluster_addr":"172.21.15.142:6803/3327577487","heartbeat_back_addr":"172.21.15.142:6807/3327577487","heartbeat_front_addr":"172.21.15.142:6805/3327577487","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.386986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.358863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.645118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.318690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.618213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.843287+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[1,0]}],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":2,"to":3}]},{"pgid":"2.4","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:18:49.750 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T15:18:49.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:18:50.051 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:50.051 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":133,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:18:49.381564+0000","last_up_change":"2024-09-21T15:15:15.737314+0000","last_in_change":"2024-09-21T15:17:42.447038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:15:15.995966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:15:21.158896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6817","nonce":3600997419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6819","nonce":3600997419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6823","nonce":3600997419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6821","nonce":3600997419}]},"public_addr":"172.21.15.67:6817/3600997419","cluster_addr":"172.21.15.67:6819/3600997419","heartbeat_back_addr":"172.21.15.67:6823/3600997419","heartbeat_front_addr":"172.21.15.67:6821/3600997419","state":["exists","up"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6809","nonce":1817390160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6811","nonce":1817390160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6815","nonce":1817390160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6813","nonce":1817390160}]},"public_addr":"172.21.15.67:6809/1817390160","cluster_addr":"172.21.15.67:6811/1817390160","heartbeat_back_addr":"172.21.15.67:6815/1817390160","heartbeat_front_addr":"172.21.15.67:6813/1817390160","state":["exists","up"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6801","nonce":3371454484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6803","nonce":3371454484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6807","nonce":3371454484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6805","nonce":3371454484}]},"public_addr":"172.21.15.67:6801/3371454484","cluster_addr":"172.21.15.67:6803/3371454484","heartbeat_back_addr":"172.21.15.67:6807/3371454484","heartbeat_front_addr":"172.21.15.67:6805/3371454484","state":["exists","up"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6809","nonce":1069930087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6811","nonce":1069930087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6815","nonce":1069930087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6813","nonce":1069930087}]},"public_addr":"172.21.15.142:6809/1069930087","cluster_addr":"172.21.15.142:6811/1069930087","heartbeat_back_addr":"172.21.15.142:6815/1069930087","heartbeat_front_addr":"172.21.15.142:6813/1069930087","state":["exists","up"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6817","nonce":3606282773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6819","nonce":3606282773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6823","nonce":3606282773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6821","nonce":3606282773}]},"public_addr":"172.21.15.142:6817/3606282773","cluster_addr":"172.21.15.142:6819/3606282773","heartbeat_back_addr":"172.21.15.142:6823/3606282773","heartbeat_front_addr":"172.21.15.142:6821/3606282773","state":["exists","up"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6801","nonce":3327577487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6803","nonce":3327577487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6807","nonce":3327577487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6805","nonce":3327577487}]},"public_addr":"172.21.15.142:6801/3327577487","cluster_addr":"172.21.15.142:6803/3327577487","heartbeat_back_addr":"172.21.15.142:6807/3327577487","heartbeat_front_addr":"172.21.15.142:6805/3327577487","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.386986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.358863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.645118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.318690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.618213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.843287+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[1,0]}],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":2,"to":3}]},{"pgid":"2.4","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:18:50.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T15:18:50.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T15:18:50.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T15:18:50.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T15:18:50.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T15:18:50.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T15:18:50.224 INFO:teuthology.orchestra.run.smithi067.stdout:51539607599 2024-09-21T15:18:50.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:18:50.235 INFO:teuthology.orchestra.run.smithi067.stdout:55834574897 2024-09-21T15:18:50.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:18:50.261 INFO:teuthology.orchestra.run.smithi067.stdout:51539607601 2024-09-21T15:18:50.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:18:50.280 INFO:teuthology.orchestra.run.smithi067.stdout:55834574895 2024-09-21T15:18:50.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:18:50.281 INFO:teuthology.orchestra.run.smithi067.stdout:55834574895 2024-09-21T15:18:50.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:18:50.287 INFO:teuthology.orchestra.run.smithi067.stdout:55834574894 2024-09-21T15:18:50.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:18:50.613 INFO:teuthology.orchestra.run.smithi067.stdout:51539607600 2024-09-21T15:18:50.631 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607601 got 51539607600 for osd.4 2024-09-21T15:18:50.667 INFO:teuthology.orchestra.run.smithi067.stdout:55834574893 2024-09-21T15:18:50.681 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574894 got 55834574893 for osd.0 2024-09-21T15:18:50.723 INFO:teuthology.orchestra.run.smithi067.stdout:55834574896 2024-09-21T15:18:50.724 INFO:teuthology.orchestra.run.smithi067.stdout:51539607598 2024-09-21T15:18:50.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.1 2024-09-21T15:18:50.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607599 got 51539607598 for osd.2 2024-09-21T15:18:50.748 INFO:teuthology.orchestra.run.smithi067.stdout:55834574894 2024-09-21T15:18:50.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574894 for osd.3 2024-09-21T15:18:50.768 INFO:teuthology.orchestra.run.smithi067.stdout:55834574894 2024-09-21T15:18:50.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574894 for osd.5 2024-09-21T15:18:51.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:18:51.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:18:51.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:18:51.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:18:51.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:18:51.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:18:52.026 INFO:teuthology.orchestra.run.smithi067.stdout:51539607600 2024-09-21T15:18:52.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607601 got 51539607600 for osd.4 2024-09-21T15:18:52.055 INFO:teuthology.orchestra.run.smithi067.stdout:55834574893 2024-09-21T15:18:52.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574894 got 55834574893 for osd.0 2024-09-21T15:18:52.123 INFO:teuthology.orchestra.run.smithi067.stdout:55834574894 2024-09-21T15:18:52.133 INFO:teuthology.orchestra.run.smithi067.stdout:55834574896 2024-09-21T15:18:52.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574894 for osd.3 2024-09-21T15:18:52.142 INFO:teuthology.orchestra.run.smithi067.stdout:51539607598 2024-09-21T15:18:52.146 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.1 2024-09-21T15:18:52.156 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607599 got 51539607598 for osd.2 2024-09-21T15:18:52.179 INFO:teuthology.orchestra.run.smithi067.stdout:55834574894 2024-09-21T15:18:52.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574894 for osd.5 2024-09-21T15:18:53.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:18:53.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:18:53.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:18:53.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:18:53.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:18:53.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:18:53.438 INFO:teuthology.orchestra.run.smithi067.stdout:55834574895 2024-09-21T15:18:53.452 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574894 got 55834574895 for osd.0 2024-09-21T15:18:53.452 DEBUG:teuthology.parallel:result is None 2024-09-21T15:18:53.485 INFO:teuthology.orchestra.run.smithi067.stdout:51539607601 2024-09-21T15:18:53.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607601 got 51539607601 for osd.4 2024-09-21T15:18:53.504 DEBUG:teuthology.parallel:result is None 2024-09-21T15:18:53.540 INFO:teuthology.orchestra.run.smithi067.stdout:55834574896 2024-09-21T15:18:53.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574896 for osd.5 2024-09-21T15:18:53.553 DEBUG:teuthology.parallel:result is None 2024-09-21T15:18:53.573 INFO:teuthology.orchestra.run.smithi067.stdout:55834574898 2024-09-21T15:18:53.574 INFO:teuthology.orchestra.run.smithi067.stdout:55834574896 2024-09-21T15:18:53.585 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574898 for osd.1 2024-09-21T15:18:53.585 DEBUG:teuthology.parallel:result is None 2024-09-21T15:18:53.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574896 for osd.3 2024-09-21T15:18:53.587 DEBUG:teuthology.parallel:result is None 2024-09-21T15:18:53.596 INFO:teuthology.orchestra.run.smithi067.stdout:51539607599 2024-09-21T15:18:53.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607599 got 51539607599 for osd.2 2024-09-21T15:18:53.615 DEBUG:teuthology.parallel:result is None 2024-09-21T15:18:53.615 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T15:18:53.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:18:53.912 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:18:53.927 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-21T15:18:51.974715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8298,"num_read_kb":163630,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1219,"ondisk_log_size":1219,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204288,"kb_used_data":9384,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":566026752,"statfs":{"total":579820584960,"available":579611394048,"internally_reserved":0,"allocated":9609216,"data_stored":4897030,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.623150"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375615+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:18:48.375615+0000","last_peered":"2024-09-21T15:18:48.375615+0000","last_clean":"2024-09-21T15:18:48.375615+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:48.375615+0000","last_undegraded":"2024-09-21T15:18:48.375615+0000","last_fullsized":"2024-09-21T15:18:48.375615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'68","reported_seq":38,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375726+0000","last_change":"2024-09-21T15:18:41.252067+0000","last_active":"2024-09-21T15:18:48.375726+0000","last_peered":"2024-09-21T15:18:48.375726+0000","last_clean":"2024-09-21T15:18:48.375726+0000","last_became_active":"2024-09-21T15:18:41.230699+0000","last_became_peered":"2024-09-21T15:18:41.230699+0000","last_unstale":"2024-09-21T15:18:48.375726+0000","last_undegraded":"2024-09-21T15:18:48.375726+0000","last_fullsized":"2024-09-21T15:18:48.375726+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:55.466987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047058700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1189,"num_read_kb":34487,"num_write":52,"num_write_kb":32837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":33685580,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"118'16","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.496675+0000","last_change":"2024-09-21T15:18:31.265275+0000","last_active":"2024-09-21T15:18:48.496675+0000","last_peered":"2024-09-21T15:18:48.496675+0000","last_clean":"2024-09-21T15:18:48.496675+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:48.496675+0000","last_undegraded":"2024-09-21T15:18:48.496675+0000","last_fullsized":"2024-09-21T15:18:48.496675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000786274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":20,"num_write":14,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"118'34","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375627+0000","last_change":"2024-09-21T15:18:31.262815+0000","last_active":"2024-09-21T15:18:48.375627+0000","last_peered":"2024-09-21T15:18:48.375627+0000","last_clean":"2024-09-21T15:18:48.375627+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:18:48.375627+0000","last_undegraded":"2024-09-21T15:18:48.375627+0000","last_fullsized":"2024-09-21T15:18:48.375627+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067175000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":16393,"num_write":25,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"118'35","reported_seq":435,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.376171+0000","last_change":"2024-09-21T15:18:47.383598+0000","last_active":"2024-09-21T15:18:48.376171+0000","last_peered":"2024-09-21T15:18:48.376171+0000","last_clean":"2024-09-21T15:18:48.376171+0000","last_became_active":"2024-09-21T15:18:47.365686+0000","last_became_peered":"2024-09-21T15:18:47.365686+0000","last_unstale":"2024-09-21T15:18:48.376171+0000","last_undegraded":"2024-09-21T15:18:48.376171+0000","last_fullsized":"2024-09-21T15:18:48.376171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:01:45.806372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038392899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":8348,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":20971596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"118'43","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370833+0000","last_change":"2024-09-21T15:18:31.263260+0000","last_active":"2024-09-21T15:18:48.370833+0000","last_peered":"2024-09-21T15:18:48.370833+0000","last_clean":"2024-09-21T15:18:48.370833+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:48.370833+0000","last_undegraded":"2024-09-21T15:18:48.370833+0000","last_fullsized":"2024-09-21T15:18:48.370833+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010214390000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":8218,"num_write":36,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"118'272","reported_seq":1971,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375978+0000","last_change":"2024-09-21T15:18:31.262627+0000","last_active":"2024-09-21T15:18:48.375978+0000","last_peered":"2024-09-21T15:18:48.375978+0000","last_clean":"2024-09-21T15:18:48.375978+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:18:48.375978+0000","last_undegraded":"2024-09-21T15:18:48.375978+0000","last_fullsized":"2024-09-21T15:18:48.375978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000406699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1860,"num_read_kb":42861,"num_write":251,"num_write_kb":52394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'358","reported_seq":1918,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370853+0000","last_change":"2024-09-21T15:18:42.234126+0000","last_active":"2024-09-21T15:18:48.370853+0000","last_peered":"2024-09-21T15:18:48.370853+0000","last_clean":"2024-09-21T15:18:48.370853+0000","last_became_active":"2024-09-21T15:18:42.233283+0000","last_became_peered":"2024-09-21T15:18:42.233283+0000","last_unstale":"2024-09-21T15:18:48.370853+0000","last_undegraded":"2024-09-21T15:18:48.370853+0000","last_fullsized":"2024-09-21T15:18:48.370853+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:08.899167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080228000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4011,"num_read_kb":36187,"num_write":209,"num_write_kb":50269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17874944,"num_keys_recovered":40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"118'276","reported_seq":655,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375686+0000","last_change":"2024-09-21T15:18:44.239084+0000","last_active":"2024-09-21T15:18:48.375686+0000","last_peered":"2024-09-21T15:18:48.375686+0000","last_clean":"2024-09-21T15:18:48.375686+0000","last_became_active":"2024-09-21T15:18:44.237737+0000","last_became_peered":"2024-09-21T15:18:44.237737+0000","last_unstale":"2024-09-21T15:18:48.375686+0000","last_undegraded":"2024-09-21T15:18:48.375686+0000","last_fullsized":"2024-09-21T15:18:48.375686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:41:56.392117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052404700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":785,"num_read_kb":17016,"num_write":350,"num_write_kb":53469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8182,"num_read_kb":163530,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1102,"ondisk_log_size":1102,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.752},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.052,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.82499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.073,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79300000000000004,"15min":0.79100000000000004},"min":{"1min":0.55300000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.204,"15min":1.204},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79600000000000004,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.03},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0129999999999999,"5min":1.212,"15min":1.212},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.103,"5min":1.208,"15min":1.208},"last":0.878}]},{"osd":4,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.292,"5min":1.629,"15min":1.629},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84699999999999998,"15min":0.86799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1599999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0069999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.014,"5min":1.333,"15min":1.333},"last":1.0289999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0269999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.119},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96699999999999997,"5min":1.133,"15min":1.133},"last":1.1919999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.145,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.96399999999999997,"5min":1.581,"15min":1.581},"last":1.0580000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.0980000000000001,"5min":1.837,"15min":1.837},"last":0.84399999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.80800000000000005,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.85999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1659999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.88500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36812,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335028,"statfs":{"total":96636764160,"available":96599068672,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.51300000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96199999999999997,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84299999999999997,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.66500000000000004,"15min":0.624},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1220000000000001,"5min":1.226,"15min":1.226},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.016,"5min":1.349,"15min":1.349},"last":0.92300000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.86099999999999999,"15min":0.85899999999999999},"min":{"1min":0.66600000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1140000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.081,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.173,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93700000000000006,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1040000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.875},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.3160000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1}]},{"osd":4,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.101,"15min":1.101},"last":0.875},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.877,"15min":0.90700000000000003},"min":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.1399999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.68400000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.163,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.90500000000000003,"15min":0.92400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.194,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.107}]}]},{"osd":4,"up_from":12,"seq":51539607601,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.74199999999999999,"15min":0.76600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1419999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.874},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.752,"15min":0.77700000000000002},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.999,"5min":1.331,"15min":1.331},"last":0.92000000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.747},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.94299999999999995,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.84299999999999997},"min":{"1min":0.55800000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1140000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79500000000000004,"15min":0.80800000000000005},"min":{"1min":0.53000000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.083,"5min":1.272,"15min":1.272},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.80700000000000005},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98399999999999999,"5min":1.298,"15min":1.298},"last":0.89200000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.81200000000000006}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.82899999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94499999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.871},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1699999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.76800000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.72699999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":1.004,"15min":1.004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69599999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83199999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.091,"5min":1.417,"15min":1.417},"last":0.745},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.123,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.79600000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.89100000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0089999999999999,"5min":1.536,"15min":1.536},"last":0.995}]},{"osd":5,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2170000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.91800000000000004,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.042,"5min":1.595,"15min":1.595},"last":0.83599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:18:53.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:18:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:18:55.083 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-21T15:18:53.975254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8298,"num_read_kb":163630,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1219,"ondisk_log_size":1219,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204288,"kb_used_data":9384,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":566026752,"statfs":{"total":579820584960,"available":579611394048,"internally_reserved":0,"allocated":9609216,"data_stored":4897030,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.621341"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375615+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:18:48.375615+0000","last_peered":"2024-09-21T15:18:48.375615+0000","last_clean":"2024-09-21T15:18:48.375615+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:48.375615+0000","last_undegraded":"2024-09-21T15:18:48.375615+0000","last_fullsized":"2024-09-21T15:18:48.375615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'68","reported_seq":38,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375726+0000","last_change":"2024-09-21T15:18:41.252067+0000","last_active":"2024-09-21T15:18:48.375726+0000","last_peered":"2024-09-21T15:18:48.375726+0000","last_clean":"2024-09-21T15:18:48.375726+0000","last_became_active":"2024-09-21T15:18:41.230699+0000","last_became_peered":"2024-09-21T15:18:41.230699+0000","last_unstale":"2024-09-21T15:18:48.375726+0000","last_undegraded":"2024-09-21T15:18:48.375726+0000","last_fullsized":"2024-09-21T15:18:48.375726+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:55.466987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047058700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1189,"num_read_kb":34487,"num_write":52,"num_write_kb":32837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":33685580,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"118'16","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.496675+0000","last_change":"2024-09-21T15:18:31.265275+0000","last_active":"2024-09-21T15:18:48.496675+0000","last_peered":"2024-09-21T15:18:48.496675+0000","last_clean":"2024-09-21T15:18:48.496675+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:48.496675+0000","last_undegraded":"2024-09-21T15:18:48.496675+0000","last_fullsized":"2024-09-21T15:18:48.496675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000786274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":20,"num_write":14,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"118'34","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375627+0000","last_change":"2024-09-21T15:18:31.262815+0000","last_active":"2024-09-21T15:18:48.375627+0000","last_peered":"2024-09-21T15:18:48.375627+0000","last_clean":"2024-09-21T15:18:48.375627+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:18:48.375627+0000","last_undegraded":"2024-09-21T15:18:48.375627+0000","last_fullsized":"2024-09-21T15:18:48.375627+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067175000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":16393,"num_write":25,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"118'35","reported_seq":435,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.376171+0000","last_change":"2024-09-21T15:18:47.383598+0000","last_active":"2024-09-21T15:18:48.376171+0000","last_peered":"2024-09-21T15:18:48.376171+0000","last_clean":"2024-09-21T15:18:48.376171+0000","last_became_active":"2024-09-21T15:18:47.365686+0000","last_became_peered":"2024-09-21T15:18:47.365686+0000","last_unstale":"2024-09-21T15:18:48.376171+0000","last_undegraded":"2024-09-21T15:18:48.376171+0000","last_fullsized":"2024-09-21T15:18:48.376171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:01:45.806372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038392899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":8348,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":20971596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"118'43","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370833+0000","last_change":"2024-09-21T15:18:31.263260+0000","last_active":"2024-09-21T15:18:48.370833+0000","last_peered":"2024-09-21T15:18:48.370833+0000","last_clean":"2024-09-21T15:18:48.370833+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:48.370833+0000","last_undegraded":"2024-09-21T15:18:48.370833+0000","last_fullsized":"2024-09-21T15:18:48.370833+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010214390000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":8218,"num_write":36,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"118'272","reported_seq":1971,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375978+0000","last_change":"2024-09-21T15:18:31.262627+0000","last_active":"2024-09-21T15:18:48.375978+0000","last_peered":"2024-09-21T15:18:48.375978+0000","last_clean":"2024-09-21T15:18:48.375978+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:18:48.375978+0000","last_undegraded":"2024-09-21T15:18:48.375978+0000","last_fullsized":"2024-09-21T15:18:48.375978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000406699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1860,"num_read_kb":42861,"num_write":251,"num_write_kb":52394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'358","reported_seq":1918,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370853+0000","last_change":"2024-09-21T15:18:42.234126+0000","last_active":"2024-09-21T15:18:48.370853+0000","last_peered":"2024-09-21T15:18:48.370853+0000","last_clean":"2024-09-21T15:18:48.370853+0000","last_became_active":"2024-09-21T15:18:42.233283+0000","last_became_peered":"2024-09-21T15:18:42.233283+0000","last_unstale":"2024-09-21T15:18:48.370853+0000","last_undegraded":"2024-09-21T15:18:48.370853+0000","last_fullsized":"2024-09-21T15:18:48.370853+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:08.899167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080228000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4011,"num_read_kb":36187,"num_write":209,"num_write_kb":50269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17874944,"num_keys_recovered":40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"118'276","reported_seq":655,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375686+0000","last_change":"2024-09-21T15:18:44.239084+0000","last_active":"2024-09-21T15:18:48.375686+0000","last_peered":"2024-09-21T15:18:48.375686+0000","last_clean":"2024-09-21T15:18:48.375686+0000","last_became_active":"2024-09-21T15:18:44.237737+0000","last_became_peered":"2024-09-21T15:18:44.237737+0000","last_unstale":"2024-09-21T15:18:48.375686+0000","last_undegraded":"2024-09-21T15:18:48.375686+0000","last_fullsized":"2024-09-21T15:18:48.375686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:41:56.392117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052404700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":785,"num_read_kb":17016,"num_write":350,"num_write_kb":53469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8182,"num_read_kb":163530,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1102,"ondisk_log_size":1102,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.752},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.052,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.82499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.073,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79300000000000004,"15min":0.79100000000000004},"min":{"1min":0.55300000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.204,"15min":1.204},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79600000000000004,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.03},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0129999999999999,"5min":1.212,"15min":1.212},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.103,"5min":1.208,"15min":1.208},"last":0.878}]},{"osd":4,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.292,"5min":1.629,"15min":1.629},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84699999999999998,"15min":0.86799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1599999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0069999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.014,"5min":1.333,"15min":1.333},"last":1.0289999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0269999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.119},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96699999999999997,"5min":1.133,"15min":1.133},"last":1.1919999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.145,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.96399999999999997,"5min":1.581,"15min":1.581},"last":1.0580000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.0980000000000001,"5min":1.837,"15min":1.837},"last":0.84399999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.80800000000000005,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.85999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1659999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.88500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36812,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335028,"statfs":{"total":96636764160,"available":96599068672,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.51300000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96199999999999997,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84299999999999997,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.66500000000000004,"15min":0.624},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1220000000000001,"5min":1.226,"15min":1.226},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.016,"5min":1.349,"15min":1.349},"last":0.92300000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.86099999999999999,"15min":0.85899999999999999},"min":{"1min":0.66600000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1140000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.081,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.173,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93700000000000006,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1040000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.875},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.3160000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1}]},{"osd":4,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.101,"15min":1.101},"last":0.875},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.877,"15min":0.90700000000000003},"min":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.1399999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.68400000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.163,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.90500000000000003,"15min":0.92400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.194,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.107}]}]},{"osd":4,"up_from":12,"seq":51539607601,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.74199999999999999,"15min":0.76600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1419999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.874},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.752,"15min":0.77700000000000002},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.999,"5min":1.331,"15min":1.331},"last":0.92000000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.747},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.94299999999999995,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.84299999999999997},"min":{"1min":0.55800000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1140000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79500000000000004,"15min":0.80800000000000005},"min":{"1min":0.53000000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.083,"5min":1.272,"15min":1.272},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.80700000000000005},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98399999999999999,"5min":1.298,"15min":1.298},"last":0.89200000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.81200000000000006}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.82899999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94499999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.871},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1699999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.76800000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.72699999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":1.004,"15min":1.004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69599999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83199999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.091,"5min":1.417,"15min":1.417},"last":0.745},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.123,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.79600000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.89100000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0089999999999999,"5min":1.536,"15min":1.536},"last":0.995}]},{"osd":5,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2170000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.91800000000000004,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.042,"5min":1.595,"15min":1.595},"last":0.83599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:18:55.084 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T15:18:55.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T15:18:55.422 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T15:18:55.435 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T15:18:55.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:18:55.748 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:18:55.761 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-21T15:18:53.975254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8298,"num_read_kb":163630,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1219,"ondisk_log_size":1219,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204288,"kb_used_data":9384,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":566026752,"statfs":{"total":579820584960,"available":579611394048,"internally_reserved":0,"allocated":9609216,"data_stored":4897030,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.621341"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375615+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:18:48.375615+0000","last_peered":"2024-09-21T15:18:48.375615+0000","last_clean":"2024-09-21T15:18:48.375615+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:48.375615+0000","last_undegraded":"2024-09-21T15:18:48.375615+0000","last_fullsized":"2024-09-21T15:18:48.375615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'68","reported_seq":38,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375726+0000","last_change":"2024-09-21T15:18:41.252067+0000","last_active":"2024-09-21T15:18:48.375726+0000","last_peered":"2024-09-21T15:18:48.375726+0000","last_clean":"2024-09-21T15:18:48.375726+0000","last_became_active":"2024-09-21T15:18:41.230699+0000","last_became_peered":"2024-09-21T15:18:41.230699+0000","last_unstale":"2024-09-21T15:18:48.375726+0000","last_undegraded":"2024-09-21T15:18:48.375726+0000","last_fullsized":"2024-09-21T15:18:48.375726+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:55.466987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047058700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1189,"num_read_kb":34487,"num_write":52,"num_write_kb":32837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":33685580,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"118'16","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.496675+0000","last_change":"2024-09-21T15:18:31.265275+0000","last_active":"2024-09-21T15:18:48.496675+0000","last_peered":"2024-09-21T15:18:48.496675+0000","last_clean":"2024-09-21T15:18:48.496675+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:48.496675+0000","last_undegraded":"2024-09-21T15:18:48.496675+0000","last_fullsized":"2024-09-21T15:18:48.496675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000786274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":20,"num_write":14,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"118'34","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375627+0000","last_change":"2024-09-21T15:18:31.262815+0000","last_active":"2024-09-21T15:18:48.375627+0000","last_peered":"2024-09-21T15:18:48.375627+0000","last_clean":"2024-09-21T15:18:48.375627+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:18:48.375627+0000","last_undegraded":"2024-09-21T15:18:48.375627+0000","last_fullsized":"2024-09-21T15:18:48.375627+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067175000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":16393,"num_write":25,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"118'35","reported_seq":435,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.376171+0000","last_change":"2024-09-21T15:18:47.383598+0000","last_active":"2024-09-21T15:18:48.376171+0000","last_peered":"2024-09-21T15:18:48.376171+0000","last_clean":"2024-09-21T15:18:48.376171+0000","last_became_active":"2024-09-21T15:18:47.365686+0000","last_became_peered":"2024-09-21T15:18:47.365686+0000","last_unstale":"2024-09-21T15:18:48.376171+0000","last_undegraded":"2024-09-21T15:18:48.376171+0000","last_fullsized":"2024-09-21T15:18:48.376171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:01:45.806372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038392899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":8348,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":20971596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"118'43","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370833+0000","last_change":"2024-09-21T15:18:31.263260+0000","last_active":"2024-09-21T15:18:48.370833+0000","last_peered":"2024-09-21T15:18:48.370833+0000","last_clean":"2024-09-21T15:18:48.370833+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:48.370833+0000","last_undegraded":"2024-09-21T15:18:48.370833+0000","last_fullsized":"2024-09-21T15:18:48.370833+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010214390000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":8218,"num_write":36,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"118'272","reported_seq":1971,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375978+0000","last_change":"2024-09-21T15:18:31.262627+0000","last_active":"2024-09-21T15:18:48.375978+0000","last_peered":"2024-09-21T15:18:48.375978+0000","last_clean":"2024-09-21T15:18:48.375978+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:18:48.375978+0000","last_undegraded":"2024-09-21T15:18:48.375978+0000","last_fullsized":"2024-09-21T15:18:48.375978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000406699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1860,"num_read_kb":42861,"num_write":251,"num_write_kb":52394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'358","reported_seq":1918,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370853+0000","last_change":"2024-09-21T15:18:42.234126+0000","last_active":"2024-09-21T15:18:48.370853+0000","last_peered":"2024-09-21T15:18:48.370853+0000","last_clean":"2024-09-21T15:18:48.370853+0000","last_became_active":"2024-09-21T15:18:42.233283+0000","last_became_peered":"2024-09-21T15:18:42.233283+0000","last_unstale":"2024-09-21T15:18:48.370853+0000","last_undegraded":"2024-09-21T15:18:48.370853+0000","last_fullsized":"2024-09-21T15:18:48.370853+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:08.899167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080228000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4011,"num_read_kb":36187,"num_write":209,"num_write_kb":50269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17874944,"num_keys_recovered":40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"118'276","reported_seq":655,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375686+0000","last_change":"2024-09-21T15:18:44.239084+0000","last_active":"2024-09-21T15:18:48.375686+0000","last_peered":"2024-09-21T15:18:48.375686+0000","last_clean":"2024-09-21T15:18:48.375686+0000","last_became_active":"2024-09-21T15:18:44.237737+0000","last_became_peered":"2024-09-21T15:18:44.237737+0000","last_unstale":"2024-09-21T15:18:48.375686+0000","last_undegraded":"2024-09-21T15:18:48.375686+0000","last_fullsized":"2024-09-21T15:18:48.375686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:41:56.392117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052404700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":785,"num_read_kb":17016,"num_write":350,"num_write_kb":53469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8182,"num_read_kb":163530,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1102,"ondisk_log_size":1102,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.752},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.052,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.82499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.073,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79300000000000004,"15min":0.79100000000000004},"min":{"1min":0.55300000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.204,"15min":1.204},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79600000000000004,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.03},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0129999999999999,"5min":1.212,"15min":1.212},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.103,"5min":1.208,"15min":1.208},"last":0.878}]},{"osd":4,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.292,"5min":1.629,"15min":1.629},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84699999999999998,"15min":0.86799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1599999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0069999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.014,"5min":1.333,"15min":1.333},"last":1.0289999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0269999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.119},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96699999999999997,"5min":1.133,"15min":1.133},"last":1.1919999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.145,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.96399999999999997,"5min":1.581,"15min":1.581},"last":1.0580000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.0980000000000001,"5min":1.837,"15min":1.837},"last":0.84399999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.80800000000000005,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.85999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1659999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.88500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36812,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335028,"statfs":{"total":96636764160,"available":96599068672,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.51300000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96199999999999997,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84299999999999997,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.66500000000000004,"15min":0.624},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1220000000000001,"5min":1.226,"15min":1.226},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.016,"5min":1.349,"15min":1.349},"last":0.92300000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.86099999999999999,"15min":0.85899999999999999},"min":{"1min":0.66600000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1140000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.081,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.173,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93700000000000006,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1040000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.875},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.3160000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1}]},{"osd":4,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.101,"15min":1.101},"last":0.875},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.877,"15min":0.90700000000000003},"min":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.1399999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.68400000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.163,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.90500000000000003,"15min":0.92400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.194,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.107}]}]},{"osd":4,"up_from":12,"seq":51539607601,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.74199999999999999,"15min":0.76600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1419999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.874},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.752,"15min":0.77700000000000002},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.999,"5min":1.331,"15min":1.331},"last":0.92000000000000004}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.747},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.94299999999999995,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.84299999999999997},"min":{"1min":0.55800000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1140000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79500000000000004,"15min":0.80800000000000005},"min":{"1min":0.53000000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.083,"5min":1.272,"15min":1.272},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.80700000000000005},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98399999999999999,"5min":1.298,"15min":1.298},"last":0.89200000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.81200000000000006}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.82899999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94499999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.871},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1699999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.76800000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.72699999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":1.004,"15min":1.004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69599999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83199999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.091,"5min":1.417,"15min":1.417},"last":0.745},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.123,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.79600000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.89100000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0089999999999999,"5min":1.536,"15min":1.536},"last":0.995}]},{"osd":5,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2170000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.91800000000000004,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.042,"5min":1.595,"15min":1.595},"last":0.83599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:18:55.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:18:56.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:56.069 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:18:56.081 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-21T15:18:55.975844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8298,"num_read_kb":163630,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1219,"ondisk_log_size":1219,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204288,"kb_used_data":9384,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":566026752,"statfs":{"total":579820584960,"available":579611394048,"internally_reserved":0,"allocated":9609216,"data_stored":4897030,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002204"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375615+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:18:48.375615+0000","last_peered":"2024-09-21T15:18:48.375615+0000","last_clean":"2024-09-21T15:18:48.375615+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:48.375615+0000","last_undegraded":"2024-09-21T15:18:48.375615+0000","last_fullsized":"2024-09-21T15:18:48.375615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'68","reported_seq":38,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375726+0000","last_change":"2024-09-21T15:18:41.252067+0000","last_active":"2024-09-21T15:18:48.375726+0000","last_peered":"2024-09-21T15:18:48.375726+0000","last_clean":"2024-09-21T15:18:48.375726+0000","last_became_active":"2024-09-21T15:18:41.230699+0000","last_became_peered":"2024-09-21T15:18:41.230699+0000","last_unstale":"2024-09-21T15:18:48.375726+0000","last_undegraded":"2024-09-21T15:18:48.375726+0000","last_fullsized":"2024-09-21T15:18:48.375726+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:55.466987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047058700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1189,"num_read_kb":34487,"num_write":52,"num_write_kb":32837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":33685580,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"118'16","reported_seq":151,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:18:49.399628+0000","last_change":"2024-09-21T15:18:31.265275+0000","last_active":"2024-09-21T15:18:49.399628+0000","last_peered":"2024-09-21T15:18:49.399628+0000","last_clean":"2024-09-21T15:18:49.399628+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:49.399628+0000","last_undegraded":"2024-09-21T15:18:49.399628+0000","last_fullsized":"2024-09-21T15:18:49.399628+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000786274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":20,"num_write":14,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"118'34","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375627+0000","last_change":"2024-09-21T15:18:31.262815+0000","last_active":"2024-09-21T15:18:48.375627+0000","last_peered":"2024-09-21T15:18:48.375627+0000","last_clean":"2024-09-21T15:18:48.375627+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:18:48.375627+0000","last_undegraded":"2024-09-21T15:18:48.375627+0000","last_fullsized":"2024-09-21T15:18:48.375627+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067175000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":16393,"num_write":25,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"118'35","reported_seq":435,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.376171+0000","last_change":"2024-09-21T15:18:47.383598+0000","last_active":"2024-09-21T15:18:48.376171+0000","last_peered":"2024-09-21T15:18:48.376171+0000","last_clean":"2024-09-21T15:18:48.376171+0000","last_became_active":"2024-09-21T15:18:47.365686+0000","last_became_peered":"2024-09-21T15:18:47.365686+0000","last_unstale":"2024-09-21T15:18:48.376171+0000","last_undegraded":"2024-09-21T15:18:48.376171+0000","last_fullsized":"2024-09-21T15:18:48.376171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:01:45.806372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038392899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":8348,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":20971596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"118'43","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370833+0000","last_change":"2024-09-21T15:18:31.263260+0000","last_active":"2024-09-21T15:18:48.370833+0000","last_peered":"2024-09-21T15:18:48.370833+0000","last_clean":"2024-09-21T15:18:48.370833+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:48.370833+0000","last_undegraded":"2024-09-21T15:18:48.370833+0000","last_fullsized":"2024-09-21T15:18:48.370833+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010214390000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":8218,"num_write":36,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"118'272","reported_seq":1971,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375978+0000","last_change":"2024-09-21T15:18:31.262627+0000","last_active":"2024-09-21T15:18:48.375978+0000","last_peered":"2024-09-21T15:18:48.375978+0000","last_clean":"2024-09-21T15:18:48.375978+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:18:48.375978+0000","last_undegraded":"2024-09-21T15:18:48.375978+0000","last_fullsized":"2024-09-21T15:18:48.375978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000406699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1860,"num_read_kb":42861,"num_write":251,"num_write_kb":52394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'358","reported_seq":1918,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370853+0000","last_change":"2024-09-21T15:18:42.234126+0000","last_active":"2024-09-21T15:18:48.370853+0000","last_peered":"2024-09-21T15:18:48.370853+0000","last_clean":"2024-09-21T15:18:48.370853+0000","last_became_active":"2024-09-21T15:18:42.233283+0000","last_became_peered":"2024-09-21T15:18:42.233283+0000","last_unstale":"2024-09-21T15:18:48.370853+0000","last_undegraded":"2024-09-21T15:18:48.370853+0000","last_fullsized":"2024-09-21T15:18:48.370853+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:08.899167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080228000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4011,"num_read_kb":36187,"num_write":209,"num_write_kb":50269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17874944,"num_keys_recovered":40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"118'276","reported_seq":655,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375686+0000","last_change":"2024-09-21T15:18:44.239084+0000","last_active":"2024-09-21T15:18:48.375686+0000","last_peered":"2024-09-21T15:18:48.375686+0000","last_clean":"2024-09-21T15:18:48.375686+0000","last_became_active":"2024-09-21T15:18:44.237737+0000","last_became_peered":"2024-09-21T15:18:44.237737+0000","last_unstale":"2024-09-21T15:18:48.375686+0000","last_undegraded":"2024-09-21T15:18:48.375686+0000","last_fullsized":"2024-09-21T15:18:48.375686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:41:56.392117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052404700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":785,"num_read_kb":17016,"num_write":350,"num_write_kb":53469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8182,"num_read_kb":163530,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1102,"ondisk_log_size":1102,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.752},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.052,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.82499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.073,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79300000000000004,"15min":0.79100000000000004},"min":{"1min":0.55300000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.204,"15min":1.204},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79600000000000004,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.03},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0129999999999999,"5min":1.212,"15min":1.212},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.103,"5min":1.208,"15min":1.208},"last":0.878}]},{"osd":4,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.292,"5min":1.629,"15min":1.629},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84699999999999998,"15min":0.86799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1599999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0069999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.014,"5min":1.333,"15min":1.333},"last":1.0289999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0269999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.119},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96699999999999997,"5min":1.133,"15min":1.133},"last":1.1919999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.145,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.96399999999999997,"5min":1.581,"15min":1.581},"last":1.0580000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.0980000000000001,"5min":1.837,"15min":1.837},"last":0.84399999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.80800000000000005,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.85999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1659999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.88500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36812,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335028,"statfs":{"total":96636764160,"available":96599068672,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.51300000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96199999999999997,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84299999999999997,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.66500000000000004,"15min":0.624},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1220000000000001,"5min":1.226,"15min":1.226},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.016,"5min":1.349,"15min":1.349},"last":0.92300000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.86099999999999999,"15min":0.85899999999999999},"min":{"1min":0.66600000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1140000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.081,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.173,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93700000000000006,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1040000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.875},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.3160000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1}]},{"osd":4,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.101,"15min":1.101},"last":0.875},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.877,"15min":0.90700000000000003},"min":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.1399999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.68400000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.163,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.90500000000000003,"15min":0.92400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.194,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.107}]}]},{"osd":4,"up_from":12,"seq":51539607602,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.74199999999999999,"15min":0.76600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1419999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.752,"15min":0.77700000000000002},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.999,"5min":1.331,"15min":1.331},"last":0.83399999999999996}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.747},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.94299999999999995,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.84299999999999997},"min":{"1min":0.55800000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1140000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.626}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79500000000000004,"15min":0.80800000000000005},"min":{"1min":0.53000000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.083,"5min":1.272,"15min":1.272},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.80700000000000005},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98399999999999999,"5min":1.298,"15min":1.298},"last":0.874}]},{"osd":3,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.82899999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94499999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.871},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1699999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0109999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607600,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.65700000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":1.004,"15min":1.004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69599999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83199999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.091,"5min":1.417,"15min":1.417},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.123,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.89100000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.873},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0089999999999999,"5min":1.536,"15min":1.536},"last":0.629}]},{"osd":5,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2170000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.91800000000000004,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.042,"5min":1.595,"15min":1.595},"last":0.77800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:18:56.083 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T15:18:56.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:18:56.389 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:18:56.403 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-21T15:18:55.975844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8298,"num_read_kb":163630,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1219,"ondisk_log_size":1219,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204288,"kb_used_data":9384,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":566026752,"statfs":{"total":579820584960,"available":579611394048,"internally_reserved":0,"allocated":9609216,"data_stored":4897030,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002204"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375615+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:18:48.375615+0000","last_peered":"2024-09-21T15:18:48.375615+0000","last_clean":"2024-09-21T15:18:48.375615+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:48.375615+0000","last_undegraded":"2024-09-21T15:18:48.375615+0000","last_fullsized":"2024-09-21T15:18:48.375615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'68","reported_seq":38,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375726+0000","last_change":"2024-09-21T15:18:41.252067+0000","last_active":"2024-09-21T15:18:48.375726+0000","last_peered":"2024-09-21T15:18:48.375726+0000","last_clean":"2024-09-21T15:18:48.375726+0000","last_became_active":"2024-09-21T15:18:41.230699+0000","last_became_peered":"2024-09-21T15:18:41.230699+0000","last_unstale":"2024-09-21T15:18:48.375726+0000","last_undegraded":"2024-09-21T15:18:48.375726+0000","last_fullsized":"2024-09-21T15:18:48.375726+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:55.466987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047058700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1189,"num_read_kb":34487,"num_write":52,"num_write_kb":32837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":33685580,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"118'16","reported_seq":151,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:18:49.399628+0000","last_change":"2024-09-21T15:18:31.265275+0000","last_active":"2024-09-21T15:18:49.399628+0000","last_peered":"2024-09-21T15:18:49.399628+0000","last_clean":"2024-09-21T15:18:49.399628+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:49.399628+0000","last_undegraded":"2024-09-21T15:18:49.399628+0000","last_fullsized":"2024-09-21T15:18:49.399628+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:18:03.794617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000786274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":20,"num_write":14,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"118'34","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375627+0000","last_change":"2024-09-21T15:18:31.262815+0000","last_active":"2024-09-21T15:18:48.375627+0000","last_peered":"2024-09-21T15:18:48.375627+0000","last_clean":"2024-09-21T15:18:48.375627+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:18:48.375627+0000","last_undegraded":"2024-09-21T15:18:48.375627+0000","last_fullsized":"2024-09-21T15:18:48.375627+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067175000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":16393,"num_write":25,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"118'35","reported_seq":435,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.376171+0000","last_change":"2024-09-21T15:18:47.383598+0000","last_active":"2024-09-21T15:18:48.376171+0000","last_peered":"2024-09-21T15:18:48.376171+0000","last_clean":"2024-09-21T15:18:48.376171+0000","last_became_active":"2024-09-21T15:18:47.365686+0000","last_became_peered":"2024-09-21T15:18:47.365686+0000","last_unstale":"2024-09-21T15:18:48.376171+0000","last_undegraded":"2024-09-21T15:18:48.376171+0000","last_fullsized":"2024-09-21T15:18:48.376171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:01:45.806372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038392899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":8348,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":20971596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"118'43","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370833+0000","last_change":"2024-09-21T15:18:31.263260+0000","last_active":"2024-09-21T15:18:48.370833+0000","last_peered":"2024-09-21T15:18:48.370833+0000","last_clean":"2024-09-21T15:18:48.370833+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:48.370833+0000","last_undegraded":"2024-09-21T15:18:48.370833+0000","last_fullsized":"2024-09-21T15:18:48.370833+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010214390000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":8218,"num_write":36,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"118'272","reported_seq":1971,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375978+0000","last_change":"2024-09-21T15:18:31.262627+0000","last_active":"2024-09-21T15:18:48.375978+0000","last_peered":"2024-09-21T15:18:48.375978+0000","last_clean":"2024-09-21T15:18:48.375978+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:18:48.375978+0000","last_undegraded":"2024-09-21T15:18:48.375978+0000","last_fullsized":"2024-09-21T15:18:48.375978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000406699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1860,"num_read_kb":42861,"num_write":251,"num_write_kb":52394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'358","reported_seq":1918,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370853+0000","last_change":"2024-09-21T15:18:42.234126+0000","last_active":"2024-09-21T15:18:48.370853+0000","last_peered":"2024-09-21T15:18:48.370853+0000","last_clean":"2024-09-21T15:18:48.370853+0000","last_became_active":"2024-09-21T15:18:42.233283+0000","last_became_peered":"2024-09-21T15:18:42.233283+0000","last_unstale":"2024-09-21T15:18:48.370853+0000","last_undegraded":"2024-09-21T15:18:48.370853+0000","last_fullsized":"2024-09-21T15:18:48.370853+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:08.899167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080228000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4011,"num_read_kb":36187,"num_write":209,"num_write_kb":50269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17874944,"num_keys_recovered":40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"118'276","reported_seq":655,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375686+0000","last_change":"2024-09-21T15:18:44.239084+0000","last_active":"2024-09-21T15:18:48.375686+0000","last_peered":"2024-09-21T15:18:48.375686+0000","last_clean":"2024-09-21T15:18:48.375686+0000","last_became_active":"2024-09-21T15:18:44.237737+0000","last_became_peered":"2024-09-21T15:18:44.237737+0000","last_unstale":"2024-09-21T15:18:48.375686+0000","last_undegraded":"2024-09-21T15:18:48.375686+0000","last_fullsized":"2024-09-21T15:18:48.375686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:41:56.392117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052404700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":785,"num_read_kb":17016,"num_write":350,"num_write_kb":53469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8182,"num_read_kb":163530,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1102,"ondisk_log_size":1102,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.752},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.052,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.82499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.073,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79300000000000004,"15min":0.79100000000000004},"min":{"1min":0.55300000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.204,"15min":1.204},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79600000000000004,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.03},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0129999999999999,"5min":1.212,"15min":1.212},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.103,"5min":1.208,"15min":1.208},"last":0.878}]},{"osd":4,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.292,"5min":1.629,"15min":1.629},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84699999999999998,"15min":0.86799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1599999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0069999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.014,"5min":1.333,"15min":1.333},"last":1.0289999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0269999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.119},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96699999999999997,"5min":1.133,"15min":1.133},"last":1.1919999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.145,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.96399999999999997,"5min":1.581,"15min":1.581},"last":1.0580000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.0980000000000001,"5min":1.837,"15min":1.837},"last":0.84399999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.80800000000000005,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.85999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1659999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.88500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36812,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335028,"statfs":{"total":96636764160,"available":96599068672,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.51300000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96199999999999997,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84299999999999997,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.74399999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.66500000000000004,"15min":0.624},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1220000000000001,"5min":1.226,"15min":1.226},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.016,"5min":1.349,"15min":1.349},"last":0.92300000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.86099999999999999,"15min":0.85899999999999999},"min":{"1min":0.66600000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1140000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.081,"5min":1.419,"15min":1.419},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.173,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.877}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93700000000000006,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1040000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.875},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.3160000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1}]},{"osd":4,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.101,"15min":1.101},"last":0.875},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.877,"15min":0.90700000000000003},"min":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.1399999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.68400000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.163,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.90500000000000003,"15min":0.92400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.194,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.107}]}]},{"osd":4,"up_from":12,"seq":51539607602,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.74199999999999999,"15min":0.76600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1419999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.752,"15min":0.77700000000000002},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.999,"5min":1.331,"15min":1.331},"last":0.83399999999999996}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.747},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.94299999999999995,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.84299999999999997},"min":{"1min":0.55800000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1140000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.626}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79500000000000004,"15min":0.80800000000000005},"min":{"1min":0.53000000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.083,"5min":1.272,"15min":1.272},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.80700000000000005},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98399999999999999,"5min":1.298,"15min":1.298},"last":0.874}]},{"osd":3,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.82899999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94499999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.871},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1699999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0109999999999999}]}]},{"osd":2,"up_from":12,"seq":51539607600,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.65700000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":1.004,"15min":1.004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69599999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83199999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.091,"5min":1.417,"15min":1.417},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.123,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.89100000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.873},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0089999999999999,"5min":1.536,"15min":1.536},"last":0.629}]},{"osd":5,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2170000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.91800000000000004,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.042,"5min":1.595,"15min":1.595},"last":0.77800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:18:56.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:18:56.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:18:56.712 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":133,"fsid":"fa0b2d62-c633-4668-b5db-f1f6328d1af9","created":"2024-09-21T15:15:04.255598+0000","modified":"2024-09-21T15:18:49.381564+0000","last_up_change":"2024-09-21T15:15:15.737314+0000","last_in_change":"2024-09-21T15:17:42.447038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:15:15.995966+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:15:21.158896+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":118,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6194913c-995a-4f8b-b2e4-1ae24c7fc1c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6817","nonce":3600997419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6819","nonce":3600997419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6823","nonce":3600997419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3600997419},{"type":"v1","addr":"172.21.15.67:6821","nonce":3600997419}]},"public_addr":"172.21.15.67:6817/3600997419","cluster_addr":"172.21.15.67:6819/3600997419","heartbeat_back_addr":"172.21.15.67:6823/3600997419","heartbeat_front_addr":"172.21.15.67:6821/3600997419","state":["exists","up"]},{"osd":1,"uuid":"496a596a-f018-4010-bd93-6df3cfbef67f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6809","nonce":1817390160}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6811","nonce":1817390160}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6815","nonce":1817390160}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1817390160},{"type":"v1","addr":"172.21.15.67:6813","nonce":1817390160}]},"public_addr":"172.21.15.67:6809/1817390160","cluster_addr":"172.21.15.67:6811/1817390160","heartbeat_back_addr":"172.21.15.67:6815/1817390160","heartbeat_front_addr":"172.21.15.67:6813/1817390160","state":["exists","up"]},{"osd":2,"uuid":"0131c1a7-80b4-448b-b7b2-89e6ba9ec584","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6801","nonce":3371454484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6803","nonce":3371454484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6807","nonce":3371454484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3371454484},{"type":"v1","addr":"172.21.15.67:6805","nonce":3371454484}]},"public_addr":"172.21.15.67:6801/3371454484","cluster_addr":"172.21.15.67:6803/3371454484","heartbeat_back_addr":"172.21.15.67:6807/3371454484","heartbeat_front_addr":"172.21.15.67:6805/3371454484","state":["exists","up"]},{"osd":3,"uuid":"edacb78d-2a42-4dd8-a648-86031fd9ad18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6809","nonce":1069930087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6811","nonce":1069930087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6815","nonce":1069930087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":1069930087},{"type":"v1","addr":"172.21.15.142:6813","nonce":1069930087}]},"public_addr":"172.21.15.142:6809/1069930087","cluster_addr":"172.21.15.142:6811/1069930087","heartbeat_back_addr":"172.21.15.142:6815/1069930087","heartbeat_front_addr":"172.21.15.142:6813/1069930087","state":["exists","up"]},{"osd":4,"uuid":"bd5f5e45-2c13-432b-9aba-69c731ca2248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6817","nonce":3606282773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6819","nonce":3606282773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6823","nonce":3606282773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":3606282773},{"type":"v1","addr":"172.21.15.142:6821","nonce":3606282773}]},"public_addr":"172.21.15.142:6817/3606282773","cluster_addr":"172.21.15.142:6819/3606282773","heartbeat_back_addr":"172.21.15.142:6823/3606282773","heartbeat_front_addr":"172.21.15.142:6821/3606282773","state":["exists","up"]},{"osd":5,"uuid":"52ceefef-20a8-41fb-ab77-053021d17105","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6801","nonce":3327577487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6803","nonce":3327577487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6807","nonce":3327577487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":3327577487},{"type":"v1","addr":"172.21.15.142:6805","nonce":3327577487}]},"public_addr":"172.21.15.142:6801/3327577487","cluster_addr":"172.21.15.142:6803/3327577487","heartbeat_back_addr":"172.21.15.142:6807/3327577487","heartbeat_front_addr":"172.21.15.142:6805/3327577487","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.386986+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.358863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.645118+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:11.318690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.618213+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:15:10.843287+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[1,0]}],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":2,"to":3}]},{"pgid":"2.4","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:18:57.728 INFO:tasks.ceph:Scrubbing osd.0 2024-09-21T15:18:57.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:18:57.891 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:57.891 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:18:57.891 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:57.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-21T15:18:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-09-21T15:18:58.231 INFO:tasks.ceph:Scrubbing osd.1 2024-09-21T15:18:58.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:18:58.379 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:58.379 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:18:58.380 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:58.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-21T15:18:58.711 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-09-21T15:18:58.725 INFO:tasks.ceph:Scrubbing osd.2 2024-09-21T15:18:58.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:18:58.878 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:18:58.878 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:58.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-21T15:18:59.199 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-09-21T15:18:59.210 INFO:tasks.ceph:Scrubbing osd.3 2024-09-21T15:18:59.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:18:59.356 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:59.356 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:18:59.356 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:59.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-21T15:18:59.687 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2024-09-21T15:18:59.698 INFO:tasks.ceph:Scrubbing osd.4 2024-09-21T15:18:59.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:18:59.851 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:18:59.851 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:18:59.851 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:18:59.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-21T15:19:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2024-09-21T15:19:00.183 INFO:tasks.ceph:Scrubbing osd.5 2024-09-21T15:19:00.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:19:00.332 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-21T15:19:00.332 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:19:00.332 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-21T15:19:00.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-21T15:19:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2024-09-21T15:19:00.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:19:00.973 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:19:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:19:00.987 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-21T15:18:59.976922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8298,"num_read_kb":163630,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1219,"ondisk_log_size":1219,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204288,"kb_used_data":9384,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":566026752,"statfs":{"total":579820584960,"available":579611394048,"internally_reserved":0,"allocated":9609216,"data_stored":4897030,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.582415"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375615+0000","last_change":"2024-09-21T15:15:17.834284+0000","last_active":"2024-09-21T15:18:48.375615+0000","last_peered":"2024-09-21T15:18:48.375615+0000","last_clean":"2024-09-21T15:18:48.375615+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:48.375615+0000","last_undegraded":"2024-09-21T15:18:48.375615+0000","last_fullsized":"2024-09-21T15:18:48.375615+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:16.751807+0000","last_clean_scrub_stamp":"2024-09-21T15:15:16.751807+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:29.720839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"118'68","reported_seq":38,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375726+0000","last_change":"2024-09-21T15:18:41.252067+0000","last_active":"2024-09-21T15:18:48.375726+0000","last_peered":"2024-09-21T15:18:48.375726+0000","last_clean":"2024-09-21T15:18:48.375726+0000","last_became_active":"2024-09-21T15:18:41.230699+0000","last_became_peered":"2024-09-21T15:18:41.230699+0000","last_unstale":"2024-09-21T15:18:48.375726+0000","last_undegraded":"2024-09-21T15:18:48.375726+0000","last_fullsized":"2024-09-21T15:18:48.375726+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:55.466987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047058700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1189,"num_read_kb":34487,"num_write":52,"num_write_kb":32837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":33685580,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"118'16","reported_seq":163,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:18:59.672517+0000","last_change":"2024-09-21T15:18:59.672947+0000","last_active":"2024-09-21T15:18:59.672517+0000","last_peered":"2024-09-21T15:18:59.672517+0000","last_clean":"2024-09-21T15:18:59.672517+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:59.672517+0000","last_undegraded":"2024-09-21T15:18:59.672517+0000","last_fullsized":"2024-09-21T15:18:59.672517+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'16","last_scrub_stamp":"2024-09-21T15:18:59.672410+0000","last_deep_scrub":"118'16","last_deep_scrub_stamp":"2024-09-21T15:18:59.672410+0000","last_clean_scrub_stamp":"2024-09-21T15:18:59.672410+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:25:31.494750+0000","scrub_duration":0.0038775509999999999,"objects_trimmed":0,"snaptrim_duration":0.000786274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":20,"num_write":14,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"118'34","reported_seq":108,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375627+0000","last_change":"2024-09-21T15:18:31.262815+0000","last_active":"2024-09-21T15:18:48.375627+0000","last_peered":"2024-09-21T15:18:48.375627+0000","last_clean":"2024-09-21T15:18:48.375627+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:18:48.375627+0000","last_undegraded":"2024-09-21T15:18:48.375627+0000","last_fullsized":"2024-09-21T15:18:48.375627+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:31.063467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067175000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":16393,"num_write":25,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"118'35","reported_seq":435,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.376171+0000","last_change":"2024-09-21T15:18:47.383598+0000","last_active":"2024-09-21T15:18:48.376171+0000","last_peered":"2024-09-21T15:18:48.376171+0000","last_clean":"2024-09-21T15:18:48.376171+0000","last_became_active":"2024-09-21T15:18:47.365686+0000","last_became_peered":"2024-09-21T15:18:47.365686+0000","last_unstale":"2024-09-21T15:18:48.376171+0000","last_undegraded":"2024-09-21T15:18:48.376171+0000","last_fullsized":"2024-09-21T15:18:48.376171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:01:45.806372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038392899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":8348,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":20971596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"118'43","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370833+0000","last_change":"2024-09-21T15:18:31.263260+0000","last_active":"2024-09-21T15:18:48.370833+0000","last_peered":"2024-09-21T15:18:48.370833+0000","last_clean":"2024-09-21T15:18:48.370833+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:18:48.370833+0000","last_undegraded":"2024-09-21T15:18:48.370833+0000","last_fullsized":"2024-09-21T15:18:48.370833+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:08:05.227782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010214390000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":8218,"num_write":36,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"118'272","reported_seq":1971,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375978+0000","last_change":"2024-09-21T15:18:31.262627+0000","last_active":"2024-09-21T15:18:48.375978+0000","last_peered":"2024-09-21T15:18:48.375978+0000","last_clean":"2024-09-21T15:18:48.375978+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:18:48.375978+0000","last_undegraded":"2024-09-21T15:18:48.375978+0000","last_fullsized":"2024-09-21T15:18:48.375978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:55.571801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000406699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1860,"num_read_kb":42861,"num_write":251,"num_write_kb":52394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'358","reported_seq":1918,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.370853+0000","last_change":"2024-09-21T15:18:42.234126+0000","last_active":"2024-09-21T15:18:48.370853+0000","last_peered":"2024-09-21T15:18:48.370853+0000","last_clean":"2024-09-21T15:18:48.370853+0000","last_became_active":"2024-09-21T15:18:42.233283+0000","last_became_peered":"2024-09-21T15:18:42.233283+0000","last_unstale":"2024-09-21T15:18:48.370853+0000","last_undegraded":"2024-09-21T15:18:48.370853+0000","last_fullsized":"2024-09-21T15:18:48.370853+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:55:08.899167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080228000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4011,"num_read_kb":36187,"num_write":209,"num_write_kb":50269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17874944,"num_keys_recovered":40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"118'276","reported_seq":655,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T15:18:48.375686+0000","last_change":"2024-09-21T15:18:44.239084+0000","last_active":"2024-09-21T15:18:48.375686+0000","last_peered":"2024-09-21T15:18:48.375686+0000","last_clean":"2024-09-21T15:18:48.375686+0000","last_became_active":"2024-09-21T15:18:44.237737+0000","last_became_peered":"2024-09-21T15:18:44.237737+0000","last_unstale":"2024-09-21T15:18:48.375686+0000","last_undegraded":"2024-09-21T15:18:48.375686+0000","last_fullsized":"2024-09-21T15:18:48.375686+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:15:21.793256+0000","last_clean_scrub_stamp":"2024-09-21T15:15:21.793256+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:41:56.392117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052404700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":785,"num_read_kb":17016,"num_write":350,"num_write_kb":53469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8182,"num_read_kb":163530,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1102,"ondisk_log_size":1102,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.752},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.052,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.93600000000000005}]},{"osd":1,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.82499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.073,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79300000000000004,"15min":0.79100000000000004},"min":{"1min":0.55300000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.204,"15min":1.204},"last":0.88300000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79600000000000004,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0129999999999999,"5min":1.212,"15min":1.212},"last":0.70599999999999996}]},{"osd":3,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.496},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.103,"5min":1.208,"15min":1.208},"last":0.77300000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.292,"5min":1.629,"15min":1.629},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84699999999999998,"15min":0.86799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1599999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.749}]}]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.014,"5min":1.333,"15min":1.333},"last":0.86499999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0269999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96699999999999997,"5min":1.133,"15min":1.133},"last":0.81200000000000006}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.145,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.96399999999999997,"5min":1.581,"15min":1.581},"last":0.60299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.0980000000000001,"5min":1.837,"15min":1.837},"last":0.84099999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.80800000000000005,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.85999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1659999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.73499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36812,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335028,"statfs":{"total":96636764160,"available":96599068672,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.51300000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96199999999999997,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.313},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84299999999999997,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.66500000000000004,"15min":0.624},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1220000000000001,"5min":1.226,"15min":1.226},"last":0.502},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.016,"5min":1.349,"15min":1.349},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.624},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.86099999999999999,"15min":0.85899999999999999},"min":{"1min":0.66600000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1140000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.081,"5min":1.419,"15min":1.419},"last":0.751},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.173,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.68300000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93700000000000006,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1040000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.875},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.3160000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.84299999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.101,"15min":1.101},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.877,"15min":0.90700000000000003},"min":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.94899999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.68400000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.163,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.875},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.90500000000000003,"15min":0.92400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.194,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.91100000000000003}]}]},{"osd":4,"up_from":12,"seq":51539607603,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.74199999999999999,"15min":0.76600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1419999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.752,"15min":0.77700000000000002},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.999,"5min":1.331,"15min":1.331},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.747},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.94299999999999995,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.84299999999999997},"min":{"1min":0.55800000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1140000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.69099999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79500000000000004,"15min":0.80800000000000005},"min":{"1min":0.53000000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.083,"5min":1.272,"15min":1.272},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.80700000000000005},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98399999999999999,"5min":1.298,"15min":1.298},"last":0.80400000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65200000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.82899999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94499999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.871},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1699999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.94199999999999995}]}]},{"osd":2,"up_from":12,"seq":51539607601,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":1.004,"15min":1.004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69599999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83199999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.091,"5min":1.417,"15min":1.417},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.123,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.78900000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.89100000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.87},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0089999999999999,"5min":1.536,"15min":1.536},"last":0.91000000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2170000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.91800000000000004,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.042,"5min":1.595,"15min":1.595},"last":0.84099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:19:00.990 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T15:15:16.751807+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=16, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.990 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T15:15:21.793256+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.991 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T15:15:21.793256+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.991 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T15:15:21.793256+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.991 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-21T15:15:21.793256+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.991 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T15:15:21.793256+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.991 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T15:15:21.793256+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.991 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T15:15:21.793256+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=15, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=18, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:19:00.991 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T15:19:20.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:19:21.305 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-21T15:19:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-09-21T15:19:21.318 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-21T15:19:19.982753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8298,"num_read_kb":163630,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1225,"ondisk_log_size":1225,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204248,"kb_used_data":9384,"kb_used_omap":9,"kb_used_meta":194806,"kb_avail":566026792,"statfs":{"total":579820584960,"available":579611435008,"internally_reserved":0,"allocated":9609216,"data_stored":4897030,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9561,"internal_metadata":199482023},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003664"},"pg_stats":[{"pgid":"1.0","version":"133'119","reported_seq":292,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:18:59.247983+0000","last_change":"2024-09-21T15:18:59.248272+0000","last_active":"2024-09-21T15:18:59.247983+0000","last_peered":"2024-09-21T15:18:59.247983+0000","last_clean":"2024-09-21T15:18:59.247983+0000","last_became_active":"2024-09-21T15:15:17.831431+0000","last_became_peered":"2024-09-21T15:15:17.831431+0000","last_unstale":"2024-09-21T15:18:59.247983+0000","last_undegraded":"2024-09-21T15:18:59.247983+0000","last_fullsized":"2024-09-21T15:18:59.247983+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'119","last_scrub_stamp":"2024-09-21T15:18:59.247886+0000","last_deep_scrub":"133'119","last_deep_scrub_stamp":"2024-09-21T15:18:59.247886+0000","last_clean_scrub_stamp":"2024-09-21T15:18:59.247886+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:35:18.912773+0000","scrub_duration":0.011059848000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"133'69","reported_seq":51,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:19:03.227029+0000","last_change":"2024-09-21T15:19:03.227276+0000","last_active":"2024-09-21T15:19:03.227029+0000","last_peered":"2024-09-21T15:19:03.227029+0000","last_clean":"2024-09-21T15:19:03.227029+0000","last_became_active":"2024-09-21T15:18:41.230699+0000","last_became_peered":"2024-09-21T15:18:41.230699+0000","last_unstale":"2024-09-21T15:19:03.227029+0000","last_undegraded":"2024-09-21T15:19:03.227029+0000","last_fullsized":"2024-09-21T15:19:03.227029+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'69","last_scrub_stamp":"2024-09-21T15:19:03.226977+0000","last_deep_scrub":"133'69","last_deep_scrub_stamp":"2024-09-21T15:19:03.226977+0000","last_clean_scrub_stamp":"2024-09-21T15:19:03.226977+0000","objects_scrubbed":1,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:12:29.205207+0000","scrub_duration":0.0073099000000000003,"objects_trimmed":0,"snaptrim_duration":0.00047058700000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1189,"num_read_kb":34487,"num_write":52,"num_write_kb":32837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":33685580,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"118'16","reported_seq":163,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:18:59.672517+0000","last_change":"2024-09-21T15:18:59.672947+0000","last_active":"2024-09-21T15:18:59.672517+0000","last_peered":"2024-09-21T15:18:59.672517+0000","last_clean":"2024-09-21T15:18:59.672517+0000","last_became_active":"2024-09-21T15:16:55.272541+0000","last_became_peered":"2024-09-21T15:16:55.272541+0000","last_unstale":"2024-09-21T15:18:59.672517+0000","last_undegraded":"2024-09-21T15:18:59.672517+0000","last_fullsized":"2024-09-21T15:18:59.672517+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'16","last_scrub_stamp":"2024-09-21T15:18:59.672410+0000","last_deep_scrub":"118'16","last_deep_scrub_stamp":"2024-09-21T15:18:59.672410+0000","last_clean_scrub_stamp":"2024-09-21T15:18:59.672410+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:25:31.494750+0000","scrub_duration":0.0038775509999999999,"objects_trimmed":0,"snaptrim_duration":0.000786274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":20,"num_write":14,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"118'34","reported_seq":120,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:19:02.206182+0000","last_change":"2024-09-21T15:19:02.206481+0000","last_active":"2024-09-21T15:19:02.206182+0000","last_peered":"2024-09-21T15:19:02.206182+0000","last_clean":"2024-09-21T15:19:02.206182+0000","last_became_active":"2024-09-21T15:17:49.913874+0000","last_became_peered":"2024-09-21T15:17:49.913874+0000","last_unstale":"2024-09-21T15:19:02.206182+0000","last_undegraded":"2024-09-21T15:19:02.206182+0000","last_fullsized":"2024-09-21T15:19:02.206182+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'34","last_scrub_stamp":"2024-09-21T15:19:02.206131+0000","last_deep_scrub":"118'34","last_deep_scrub_stamp":"2024-09-21T15:19:02.206131+0000","last_clean_scrub_stamp":"2024-09-21T15:19:02.206131+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:48.061943+0000","scrub_duration":0.0034971379999999999,"objects_trimmed":0,"snaptrim_duration":0.00067175000000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":16393,"num_write":25,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"133'36","reported_seq":448,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:19:00.053027+0000","last_change":"2024-09-21T15:19:00.053319+0000","last_active":"2024-09-21T15:19:00.053027+0000","last_peered":"2024-09-21T15:19:00.053027+0000","last_clean":"2024-09-21T15:19:00.053027+0000","last_became_active":"2024-09-21T15:18:47.365686+0000","last_became_peered":"2024-09-21T15:18:47.365686+0000","last_unstale":"2024-09-21T15:19:00.053027+0000","last_undegraded":"2024-09-21T15:19:00.053027+0000","last_fullsized":"2024-09-21T15:19:00.053027+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'36","last_scrub_stamp":"2024-09-21T15:19:00.052962+0000","last_deep_scrub":"133'36","last_deep_scrub_stamp":"2024-09-21T15:19:00.052962+0000","last_clean_scrub_stamp":"2024-09-21T15:19:00.052962+0000","objects_scrubbed":1,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:42.926086+0000","scrub_duration":0.0058358439999999998,"objects_trimmed":0,"snaptrim_duration":0.00038392899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":176,"num_read_kb":8348,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":20971596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"133'45","reported_seq":199,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:19:05.745860+0000","last_change":"2024-09-21T15:19:05.746117+0000","last_active":"2024-09-21T15:19:05.745860+0000","last_peered":"2024-09-21T15:19:05.745860+0000","last_clean":"2024-09-21T15:19:05.745860+0000","last_became_active":"2024-09-21T15:17:43.479414+0000","last_became_peered":"2024-09-21T15:17:43.479414+0000","last_unstale":"2024-09-21T15:19:05.745860+0000","last_undegraded":"2024-09-21T15:19:05.745860+0000","last_fullsized":"2024-09-21T15:19:05.745860+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'45","last_scrub_stamp":"2024-09-21T15:19:05.745804+0000","last_deep_scrub":"133'45","last_deep_scrub_stamp":"2024-09-21T15:19:05.745804+0000","last_clean_scrub_stamp":"2024-09-21T15:19:05.745804+0000","objects_scrubbed":2,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:02:14.554964+0000","scrub_duration":0.0070564410000000001,"objects_trimmed":0,"snaptrim_duration":0.0010214390000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":8218,"num_write":36,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":75497472,"num_keys_recovered":9,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"118'272","reported_seq":1983,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:19:00.226350+0000","last_change":"2024-09-21T15:19:00.226677+0000","last_active":"2024-09-21T15:19:00.226350+0000","last_peered":"2024-09-21T15:19:00.226350+0000","last_clean":"2024-09-21T15:19:00.226350+0000","last_became_active":"2024-09-21T15:17:24.300806+0000","last_became_peered":"2024-09-21T15:17:24.300806+0000","last_unstale":"2024-09-21T15:19:00.226350+0000","last_undegraded":"2024-09-21T15:19:00.226350+0000","last_fullsized":"2024-09-21T15:19:00.226350+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'272","last_scrub_stamp":"2024-09-21T15:19:00.226301+0000","last_deep_scrub":"118'272","last_deep_scrub_stamp":"2024-09-21T15:19:00.226301+0000","last_clean_scrub_stamp":"2024-09-21T15:19:00.226301+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:52:27.787623+0000","scrub_duration":0.003209174,"objects_trimmed":0,"snaptrim_duration":0.000406699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1860,"num_read_kb":42861,"num_write":251,"num_write_kb":52394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":20971574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"118'358","reported_seq":1931,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:19:04.770228+0000","last_change":"2024-09-21T15:19:04.770228+0000","last_active":"2024-09-21T15:19:04.770228+0000","last_peered":"2024-09-21T15:19:04.770228+0000","last_clean":"2024-09-21T15:19:04.770228+0000","last_became_active":"2024-09-21T15:18:42.233283+0000","last_became_peered":"2024-09-21T15:18:42.233283+0000","last_unstale":"2024-09-21T15:19:04.770228+0000","last_undegraded":"2024-09-21T15:19:04.770228+0000","last_fullsized":"2024-09-21T15:19:04.770228+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'358","last_scrub_stamp":"2024-09-21T15:19:04.769767+0000","last_deep_scrub":"118'358","last_deep_scrub_stamp":"2024-09-21T15:19:04.769767+0000","last_clean_scrub_stamp":"2024-09-21T15:19:04.769767+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:29.081058+0000","scrub_duration":0.0041102220000000002,"objects_trimmed":0,"snaptrim_duration":0.00080228000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4011,"num_read_kb":36187,"num_write":209,"num_write_kb":50269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17874944,"num_keys_recovered":40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"118'276","reported_seq":667,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-09-21T15:19:06.158921+0000","last_change":"2024-09-21T15:19:06.159204+0000","last_active":"2024-09-21T15:19:06.158921+0000","last_peered":"2024-09-21T15:19:06.158921+0000","last_clean":"2024-09-21T15:19:06.158921+0000","last_became_active":"2024-09-21T15:18:44.237737+0000","last_became_peered":"2024-09-21T15:18:44.237737+0000","last_unstale":"2024-09-21T15:19:06.158921+0000","last_undegraded":"2024-09-21T15:19:06.158921+0000","last_fullsized":"2024-09-21T15:19:06.158921+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'276","last_scrub_stamp":"2024-09-21T15:19:06.158872+0000","last_deep_scrub":"118'276","last_deep_scrub_stamp":"2024-09-21T15:19:06.158872+0000","last_clean_scrub_stamp":"2024-09-21T15:19:06.158872+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:58.238467+0000","scrub_duration":0.0035707930000000001,"objects_trimmed":0,"snaptrim_duration":0.00052404700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":785,"num_read_kb":17016,"num_write":350,"num_write_kb":53469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8182,"num_read_kb":163530,"num_write":964,"num_write_kb":270902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":206749956,"num_keys_recovered":49,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1106,"ondisk_log_size":1106,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11690,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32712,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.752},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.77500000000000002,"15min":0.78900000000000003},"min":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.052,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.69799999999999995}]},{"osd":1,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81799999999999995,"15min":0.82499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.073,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79300000000000004,"15min":0.79100000000000004},"min":{"1min":0.55300000000000005,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.006,"5min":1.204,"15min":1.204},"last":0.97299999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.79600000000000004,"15min":0.81999999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.80600000000000005,"15min":0.82999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0129999999999999,"5min":1.212,"15min":1.212},"last":0.91600000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84099999999999997,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.103,"5min":1.208,"15min":1.208},"last":0.84299999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80200000000000005,"15min":0.82799999999999996},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.292,"5min":1.629,"15min":1.629},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84699999999999998,"15min":0.86799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.1599999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.82299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.5,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.873},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.79300000000000004,"15min":0.82099999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.014,"5min":1.333,"15min":1.333},"last":0.77200000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0269999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96699999999999997,"5min":1.133,"15min":1.133},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81499999999999995},"min":{"1min":0.54100000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.145,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.82399999999999995,"15min":0.84499999999999997},"min":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.96399999999999997,"5min":1.581,"15min":1.581},"last":0.91900000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93100000000000005,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80200000000000005,"15min":0.80800000000000005},"min":{"1min":0.47599999999999998,"5min":0.439,"15min":0.439},"max":{"1min":1.0980000000000001,"5min":1.837,"15min":1.837},"last":0.58699999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.80800000000000005,"15min":0.81299999999999994},"min":{"1min":0.54100000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.85999999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1659999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.66500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36808,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335032,"statfs":{"total":96636764160,"available":96599072768,"internally_reserved":0,"allocated":1638400,"data_stored":1209744,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53800000000000003,"15min":0.51300000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.96199999999999997,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.505},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.56599999999999995},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.84299999999999997,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.66500000000000004,"15min":0.624},"min":{"1min":0.44,"5min":0.378,"15min":0.378},"max":{"1min":2.8839999999999999,"5min":2.8839999999999999,"15min":2.8839999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77100000000000002,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.80000000000000004,"15min":0.82399999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1220000000000001,"5min":1.226,"15min":1.226},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.82699999999999996,"15min":0.82099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.016,"5min":1.349,"15min":1.349},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1719999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.86099999999999999,"15min":0.85899999999999999},"min":{"1min":0.66600000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1140000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.65400000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.90600000000000003,"15min":0.90500000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.081,"5min":1.419,"15min":1.419},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.173,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.82999999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.56000000000000005,"15min":0.58599999999999997},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79900000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.60799999999999998,"15min":0.629},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.84599999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59299999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93700000000000006,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.81000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.1040000000000001,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.875},"min":{"1min":0.51800000000000002,"5min":0.495,"15min":0.495},"max":{"1min":1.3160000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.89100000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79900000000000004,"15min":0.80700000000000005},"min":{"1min":0.60599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.101,"15min":1.101},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.877,"15min":0.90700000000000003},"min":{"1min":0.56200000000000006,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.96199999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:18:25 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.89200000000000002,"15min":0.91800000000000004},"min":{"1min":0.68400000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.163,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.90500000000000003,"15min":0.92400000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.194,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.86299999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607607,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1585152,"data_stored":619395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.74199999999999999,"15min":0.76600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1419999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.752,"15min":0.77700000000000002},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.999,"5min":1.331,"15min":1.331},"last":0.93899999999999995}]},{"osd":1,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.747},"min":{"1min":0.57199999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.94299999999999995,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.81799999999999995,"15min":0.84299999999999997},"min":{"1min":0.55800000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1140000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.86299999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79500000000000004,"15min":0.80800000000000005},"min":{"1min":0.53000000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.083,"5min":1.272,"15min":1.272},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.80700000000000005},"min":{"1min":0.56000000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98399999999999999,"5min":1.298,"15min":1.298},"last":0.82099999999999995}]},{"osd":3,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.49399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.755,"15min":0.77900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:18:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.82899999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.94499999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84499999999999997,"15min":0.871},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1699999999999999,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":12,"seq":51539607605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1581056,"data_stored":619376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.83399999999999996}]},{"osd":1,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":1.004,"15min":1.004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.67800000000000005,"15min":0.69599999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.97499999999999998,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.66700000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83199999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.091,"5min":1.417,"15min":1.417},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.86399999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.123,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.876}]},{"osd":4,"last update":"Sat Sep 21 15:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.89100000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86299999999999999},"min":{"1min":0.65100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0089999999999999,"5min":1.536,"15min":1.536},"last":1.069}]},{"osd":5,"last update":"Sat Sep 21 15:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.93200000000000005,"15min":0.95799999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2170000000000001,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.91800000000000004,"15min":0.94499999999999995},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.042,"5min":1.595,"15min":1.595},"last":1.012}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5845,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:19:21.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T15:19:21.652 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T15:19:21.652 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T15:19:21.652 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T15:19:21.652 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:27.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-21T15:19:27.754 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T15:19:27.755 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T15:19:27.755 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:27.840 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T15:19:27.840 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T15:19:27.840 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:33.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-21T15:19:33.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-21T15:19:33.942 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T15:19:33.943 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-21T15:19:33.943 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:39.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-21T15:19:39.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-21T15:19:40.045 INFO:tasks.ceph.osd.3:Stopped 2024-09-21T15:19:40.045 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-21T15:19:40.045 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:40.131 INFO:tasks.ceph.osd.4:Stopped 2024-09-21T15:19:40.132 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-21T15:19:40.132 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:40.227 INFO:tasks.ceph.osd.5:Stopped 2024-09-21T15:19:40.227 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T15:19:40.227 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T15:19:40.227 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:40.256 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T15:19:40.256 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T15:19:40.256 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T15:19:40.256 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:40.294 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T15:19:40.295 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-21T15:19:40.295 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:40.340 INFO:tasks.ceph.mon.c:Stopped 2024-09-21T15:19:40.340 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-21T15:19:40.340 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:19:40.406 INFO:tasks.ceph.mon.b:Stopped 2024-09-21T15:19:40.406 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T15:19:40.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2024-09-21T15:19:40.427 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:19:40.427 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T15:19:40.694 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:19:40.694 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T15:19:40.960 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:19:40.960 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T15:19:41.304 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:19:41.304 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-21T15:19:41.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:19:41.568 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-21T15:19:41.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:19:41.813 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-21T15:19:41.988 INFO:tasks.ceph:Archiving mon data... 2024-09-21T15:19:41.989 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/data/mon.a.tgz 2024-09-21T15:19:41.990 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-21T15:19:41.996 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.2EYRfqcMbS 2024-09-21T15:19:41.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.2EYRfqcMbS 2024-09-21T15:19:42.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.2EYRfqcMbS 2024-09-21T15:19:42.152 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.2EYRfqcMbS is 320KB 2024-09-21T15:19:42.216 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.2EYRfqcMbS 2024-09-21T15:19:42.223 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/data/mon.c.tgz 2024-09-21T15:19:42.223 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-09-21T15:19:42.229 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.WAL6gSOWc4 2024-09-21T15:19:42.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.WAL6gSOWc4 2024-09-21T15:19:42.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.WAL6gSOWc4 2024-09-21T15:19:42.372 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.WAL6gSOWc4 is 437KB 2024-09-21T15:19:42.391 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.WAL6gSOWc4 2024-09-21T15:19:42.398 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/data/mon.b.tgz 2024-09-21T15:19:42.398 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-09-21T15:19:42.405 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.VVoI5qm5vH 2024-09-21T15:19:42.405 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VVoI5qm5vH 2024-09-21T15:19:42.519 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.VVoI5qm5vH 2024-09-21T15:19:42.539 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.VVoI5qm5vH is 437KB 2024-09-21T15:19:42.602 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.VVoI5qm5vH 2024-09-21T15:19:42.609 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T15:19:42.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T15:19:42.611 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T15:19:42.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T15:19:42.629 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-21T15:19:42.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-21T15:19:42.632 DEBUG:teuthology.orchestra.run.smithi142:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-21T15:19:42.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-21T15:19:42.654 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-21T15:19:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-21T15:19:42.657 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-21T15:19:42.663 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T15:19:42.665 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi067/crash 2024-09-21T15:19:42.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T15:19:42.680 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi142/crash 2024-09-21T15:19:42.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T15:19:42.696 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi146/crash 2024-09-21T15:19:42.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T15:19:42.714 INFO:tasks.ceph:Compressing logs... 2024-09-21T15:19:42.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:19:42.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:19:42.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:19:44.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-21T15:19:44.229 INFO:tasks.ceph:Archiving logs... 2024-09-21T15:19:44.230 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi067/log 2024-09-21T15:19:44.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T15:19:44.637 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi142/log 2024-09-21T15:19:44.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T15:19:44.781 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi146/log 2024-09-21T15:19:44.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T15:19:44.982 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T15:19:45.001 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T15:19:45.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T15:19:45.004 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T15:19:45.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T15:19:45.100 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T15:19:45.112 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T15:19:45.112 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:19:45.115 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:19:45.117 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:19:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T15:19:45.137 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-09-21T15:19:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi 67.205.162.81 3 u 28 64 177 0.071 -1.987 0.940 2024-09-21T15:19:45.137 INFO:teuthology.orchestra.run.smithi067.stdout:*hv02.front.sepi 96.245.170.99 3 u 40 64 177 0.083 2.375 2.362 2024-09-21T15:19:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi 74.6.168.72 3 u 36 64 177 0.089 -2.516 0.254 2024-09-21T15:19:45.137 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:19:45.139 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T15:19:45.140 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2024-09-21T15:19:45.140 INFO:teuthology.orchestra.run.smithi142.stdout:*hv01.front.sepi 67.205.162.81 3 u 27 64 177 0.053 1.416 1.361 2024-09-21T15:19:45.140 INFO:teuthology.orchestra.run.smithi142.stdout:+hv02.front.sepi 96.245.170.99 3 u 36 64 177 0.085 3.133 1.225 2024-09-21T15:19:45.140 INFO:teuthology.orchestra.run.smithi142.stdout:+hv03.front.sepi 74.6.168.72 3 u 28 64 177 0.094 0.034 0.987 2024-09-21T15:19:45.140 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:19:45.141 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T15:19:45.141 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-09-21T15:19:45.141 INFO:teuthology.orchestra.run.smithi146.stdout:+hv01.front.sepi 67.205.162.81 3 u 34 64 177 0.101 2.176 1.561 2024-09-21T15:19:45.141 INFO:teuthology.orchestra.run.smithi146.stdout:+hv02.front.sepi 96.245.170.99 3 u 29 64 177 0.097 3.784 1.272 2024-09-21T15:19:45.142 INFO:teuthology.orchestra.run.smithi146.stdout:*hv03.front.sepi 74.6.168.72 3 u 34 64 177 0.092 -0.561 1.091 2024-09-21T15:19:45.142 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-21T15:19:45.142 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T15:19:45.153 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T15:19:45.153 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T15:19:45.216 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T15:19:45.244 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T15:19:45.274 INFO:teuthology.task.internal:Duration was 773.528897 seconds 2024-09-21T15:19:45.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T15:19:45.289 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T15:19:45.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T15:19:45.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T15:19:45.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T15:19:45.344 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T15:19:45.344 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-09-21T15:19:45.344 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T15:19:45.356 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2024-09-21T15:19:45.357 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T15:19:45.371 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-09-21T15:19:45.371 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T15:19:45.387 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T15:19:45.387 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:19:45.389 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:19:45.391 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:19:45.469 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T15:19:45.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T15:19:45.475 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T15:19:45.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T15:19:45.561 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T15:19:45.578 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T15:19:45.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T15:19:45.595 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T15:19:45.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T15:19:45.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T15:19:45.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:19:45.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:19:45.651 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-09-21T15:19:45.654 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:19:45.666 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2024-09-21T15:19:45.683 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-09-21T15:19:45.699 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:19:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:19:45.727 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:19:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:19:45.733 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:19:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:19:45.739 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T15:19:45.748 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T15:19:45.749 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi067 2024-09-21T15:19:45.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T15:19:45.776 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi142 2024-09-21T15:19:45.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T15:19:45.801 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909971/remote/smithi146 2024-09-21T15:19:45.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T15:19:45.825 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T15:19:45.825 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T15:19:45.827 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T15:19:45.831 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T15:19:45.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T15:19:45.881 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T15:19:45.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T15:19:45.908 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T15:19:45.908 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T15:19:45.911 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T15:19:45.913 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T15:19:45.916 INFO:teuthology.orchestra.run.smithi067.stdout: 1048593 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 15:19 /home/ubuntu/cephtest 2024-09-21T15:19:45.919 INFO:teuthology.orchestra.run.smithi142.stdout: 1048593 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 15:19 /home/ubuntu/cephtest 2024-09-21T15:19:45.921 INFO:teuthology.orchestra.run.smithi146.stdout: 1048593 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 15:19 /home/ubuntu/cephtest 2024-09-21T15:19:45.923 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T15:19:45.944 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T15:19:45.989 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/rbd_nbd} duration: 773.5288965702057 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-21T15:19:45.989 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T15:19:46.083 INFO:teuthology.run:pass